Welcome to WebmasterWorld Guest from 188.8.131.52 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
nofollow noindex or disallow? Gemini23
In the Robots.txt I had placed "Disallow: /abcfolder/"
and thought that would stop Google from spidering all of the files/pages within the folder of /abcfolder/
but it doesn't seem to have worked as they are appearing in my website AW stats
I have a file (which is a php global page template with errors in it) that I don't want spidered.
it is essentially /abcfolder/xyx.php
what syntax do I use to stop it being spidered? and/or indexed?
edited by: Gemini23 at 8:21 pm (utc) on Dec. 5, 2008]
First off. Welcome to the gang. Add a NOINDEX to the page and NOFOLLOW on links that might point to the page. jimbeetle
Gemini23, that Disallow: /abcfolder/ *should* have blocked all the files in that directory. Did you have a User-agent line in the file?
And if you want to block all bots use the wildcard:
If you do decide to use the meta robots instead...
<meta name="robots" content="noindex">
...and do not block the file in robots.txt. This is so bots will be able to see and read the noindex.
I have had Google crawl and index folders which they should have known were banned via the robots.txt - it tested as successfully blocked in the WMT test robots widget - and yet they were indexing. I used WMT to remove the folder - it vanished within a couple of days and hasn't come back (some months). So - go to WMT, use the test thing to check you've correctly written the robots.txt If you haven't fix it. Then do a remove request, and you should be right :)