Sitemaps, Meta Data, and robots.txt Forum

It must be in the root directory of the web site as spiders will not look for it anywhere else

To exclude all robots from parts of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /misc/sitestats/

Exclude a specific spider from parts of the server
User-agent:slurp.so/
Disallow: /cgi-bin/
Disallow: /secure/
Disallow: /products/
Disallow:/misc/sitestats/

This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:

To allow a single robot complete access and exclude all others
User-agent: Googlebot/1.0
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

Hope this helps

The site gave me errors when I only tried the directory

/this
/and/this

but when I use a full link, it comes back as 'okay':

http//www.site.tld/this
http//www.site.tld/and/this

I have found that most bots don't listen anyways including Yahoo...they keep going after disallowed.

I'm so disgruntled with the net that I'm going to make my site go stealth.

What the hell is the point for search engines posting how to deal with their bots/spiders if they can't even control them properly?

Disallow: /blocked/
not
Disallow /blocked/