Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Robots how to

Need help



7:40 am on Mar 28, 2001 (gmt 0)

I`m new here and I need help whit the robots.txt
creation.(I`m a pure beginer)
Please can you tell me if when I disallow some directiry or file to googlebot if then are stoped
all google spiders or have I to write the full names of all hunderts of google spiders.
Thangs Maatrix


3:59 am on Mar 29, 2001 (gmt 0)

10+ Year Member

Welcome to WebmasterWorld maatrix!

As far as I know, you only have to use Googlebot. Now if the pages you put in your robots.txt are already indexed by Google, it may take a few months for Google to take them out.

User-agent: Googlebot
Disallow: /cgi-bin
Disallow: /images
Disallow: /doh.htm

This should keep Googlebot away from those parts of your site. If you want to keep all bots away from the same parts of your site, you can do this instead:

User-agent: *
Disallow: /cgi-bin
Disallow: /images
Disallow: /doh.htm

Here are some helpful links:


7:30 am on Mar 29, 2001 (gmt 0)


Featured Threads

Hot Threads This Week

Hot Threads This Month