Welcome to WebmasterWorld Guest from 107.21.163.40

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Robots how to

Need help

     

maatrix

7:40 am on Mar 28, 2001 (gmt 0)

Inactive Member
Account Expired

 
 


Hi,
I`m new here and I need help whit the robots.txt
creation.(I`m a pure beginer)
Please can you tell me if when I disallow some directiry or file to googlebot if then are stoped
all google spiders or have I to write the full names of all hunderts of google spiders.
Thangs Maatrix
3:59 am on Mar 29, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 1, 2001
posts:183
votes: 0


Welcome to WebmasterWorld maatrix!

As far as I know, you only have to use Googlebot. Now if the pages you put in your robots.txt are already indexed by Google, it may take a few months for Google to take them out.

User-agent: Googlebot
Disallow: /cgi-bin
Disallow: /images
Disallow: /doh.htm

This should keep Googlebot away from those parts of your site. If you want to keep all bots away from the same parts of your site, you can do this instead:

User-agent: *
Disallow: /cgi-bin
Disallow: /images
Disallow: /doh.htm

Here are some helpful links:
[info.webcrawler.com...]
[info.webcrawler.com...]
[webmasterworld.com...]

maatrix

7:30 am on Mar 29, 2001 (gmt 0)

Inactive Member
Account Expired

 
 


Thangs