nofollow noindex or disallow?

[Gemini23]

Hi,

In the Robots.txt I had placed "Disallow: /abcfolder/"

and thought that would stop Google from spidering all of the files/pages within the folder of /abcfolder/
but it doesn't seem to have worked as they are appearing in my website AW stats

I have a file (which is a php global page template with errors in it) that I don't want spidered.

it is essentially /abcfolder/xyx.php

what syntax do I use to stop it being spidered? and/or indexed?

Thanks :)

[edited by: Gemini23 at 8:21 pm (utc) on Dec. 5, 2008]

[Propools]

First off. Welcome to the gang. Add a NOINDEX to the page and NOFOLLOW on links that might point to the page.

[jimbeetle]

Gemini23, that Disallow: /abcfolder/ *should* have blocked all the files in that directory. Did you have a User-agent line in the file?

User-agent: Googlebot
Disallow: /abcfolder/

And if you want to block all bots use the wildcard:

User-agent: *
Disallow: /abcfolder/

If you do decide to use the meta robots instead...

<meta name="robots" content="noindex">

...and do not block the file in robots.txt. This is so bots will be able to see and read the noindex.

[leadegroot]

I have had Google crawl and index folders which they should have known were banned via the robots.txt - it tested as successfully blocked in the WMT test robots widget - and yet they were indexing.
I used WMT to remove the folder - it vanished within a couple of days and hasn't come back (some months).
So - go to WMT, use the test thing to check you've correctly written the robots.txt
If you haven't fix it.
Then do a remove request, and you should be right :)