Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

nofollow noindex or disallow?

8:20 pm on Dec 5, 2008 (gmt 0)

Preferred Member

5+ Year Member Top Contributors Of The Month

joined:Aug 30, 2007
votes: 3


In the Robots.txt I had placed "Disallow: /abcfolder/"

and thought that would stop Google from spidering all of the files/pages within the folder of /abcfolder/
but it doesn't seem to have worked as they are appearing in my website AW stats

I have a file (which is a php global page template with errors in it) that I don't want spidered.

it is essentially /abcfolder/xyx.php

what syntax do I use to stop it being spidered? and/or indexed?

Thanks :)

[edited by: Gemini23 at 8:21 pm (utc) on Dec. 5, 2008]

8:34 pm on Dec 5, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 8, 2004
votes: 1

First off. Welcome to the gang. Add a NOINDEX to the page and NOFOLLOW on links that might point to the page.
9:12 pm on Dec 5, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 26, 2002
votes: 6

Gemini23, that Disallow: /abcfolder/ *should* have blocked all the files in that directory. Did you have a User-agent line in the file?

User-agent: Googlebot
Disallow: /abcfolder/

And if you want to block all bots use the wildcard:

User-agent: *
Disallow: /abcfolder/

If you do decide to use the meta robots instead...

<meta name="robots" content="noindex">

...and do not block the file in robots.txt. This is so bots will be able to see and read the noindex.

12:09 am on Dec 6, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 27, 2003
posts: 1642
votes: 0

I have had Google crawl and index folders which they should have known were banned via the robots.txt - it tested as successfully blocked in the WMT test robots widget - and yet they were indexing.
I used WMT to remove the folder - it vanished within a couple of days and hasn't come back (some months).
So - go to WMT, use the test thing to check you've correctly written the robots.txt
If you haven't fix it.
Then do a remove request, and you should be right :)