Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

nofollow noindex or disallow?



8:20 pm on Dec 5, 2008 (gmt 0)

5+ Year Member


In the Robots.txt I had placed "Disallow: /abcfolder/"

and thought that would stop Google from spidering all of the files/pages within the folder of /abcfolder/
but it doesn't seem to have worked as they are appearing in my website AW stats

I have a file (which is a php global page template with errors in it) that I don't want spidered.

it is essentially /abcfolder/xyx.php

what syntax do I use to stop it being spidered? and/or indexed?

Thanks :)

[edited by: Gemini23 at 8:21 pm (utc) on Dec. 5, 2008]


8:34 pm on Dec 5, 2008 (gmt 0)

10+ Year Member

First off. Welcome to the gang. Add a NOINDEX to the page and NOFOLLOW on links that might point to the page.


9:12 pm on Dec 5, 2008 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Gemini23, that Disallow: /abcfolder/ *should* have blocked all the files in that directory. Did you have a User-agent line in the file?

User-agent: Googlebot
Disallow: /abcfolder/

And if you want to block all bots use the wildcard:

User-agent: *
Disallow: /abcfolder/

If you do decide to use the meta robots instead...

<meta name="robots" content="noindex">

...and do not block the file in robots.txt. This is so bots will be able to see and read the noindex.


12:09 am on Dec 6, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I have had Google crawl and index folders which they should have known were banned via the robots.txt - it tested as successfully blocked in the WMT test robots widget - and yet they were indexing.
I used WMT to remove the folder - it vanished within a couple of days and hasn't come back (some months).
So - go to WMT, use the test thing to check you've correctly written the robots.txt
If you haven't fix it.
Then do a remove request, and you should be right :)

Featured Threads

Hot Threads This Week

Hot Threads This Month