homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

nofollow noindex or disallow?

 8:20 pm on Dec 5, 2008 (gmt 0)


In the Robots.txt I had placed "Disallow: /abcfolder/"

and thought that would stop Google from spidering all of the files/pages within the folder of /abcfolder/
but it doesn't seem to have worked as they are appearing in my website AW stats

I have a file (which is a php global page template with errors in it) that I don't want spidered.

it is essentially /abcfolder/xyx.php

what syntax do I use to stop it being spidered? and/or indexed?

Thanks :)

[edited by: Gemini23 at 8:21 pm (utc) on Dec. 5, 2008]



 8:34 pm on Dec 5, 2008 (gmt 0)

First off. Welcome to the gang. Add a NOINDEX to the page and NOFOLLOW on links that might point to the page.


 9:12 pm on Dec 5, 2008 (gmt 0)

Gemini23, that Disallow: /abcfolder/ *should* have blocked all the files in that directory. Did you have a User-agent line in the file?

User-agent: Googlebot
Disallow: /abcfolder/

And if you want to block all bots use the wildcard:

User-agent: *
Disallow: /abcfolder/

If you do decide to use the meta robots instead...

<meta name="robots" content="noindex">

...and do not block the file in robots.txt. This is so bots will be able to see and read the noindex.


 12:09 am on Dec 6, 2008 (gmt 0)

I have had Google crawl and index folders which they should have known were banned via the robots.txt - it tested as successfully blocked in the WMT test robots widget - and yet they were indexing.
I used WMT to remove the folder - it vanished within a couple of days and hasn't come back (some months).
So - go to WMT, use the test thing to check you've correctly written the robots.txt
If you haven't fix it.
Then do a remove request, and you should be right :)

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved