Different Googlebots

What is the difference between the googlebots?

[halloerstmal]

Hi,
my site

has been crawled by
crawler10.googlebot.com
crawler11.googlebot.com
crawler12.googlebot.com

for 3 weeks now. However, I am not yet indexed by google.
How can I check if my complete site has been crawled? I am afraid that only the index is searched.
Formerly my robots.txt was

User-agent: *
Disallow:

I got the advice to remove the disallow line. Is this OK?
What is the difference between these bots?
Thanks for any input

[edited by: WebGuerrilla at 6:23 pm (utc) on June 13, 2003]
[edit reason] no specifics please [/edit]

[nancyb]

Welcome to WebmasterWorld halloerstmal

There are lots of threads here about the difference in googlebots and robots.txt files. Try the site search at the top of this screen.

This thread [webmasterworld.com] will get you started. Notice that disallow means nothing unless followed by a "/". If you don't want to disallow any bots you can dismiss using a robots.txt file as the default is allow all.

[Netizen]

As nancyb said, just remove the robots.txt file completely, you don't need it. I think removing the disallow line is wrong.

To find out if your complete site has been crawled you can check the web server log files for requests by crawler* and see what they've been munching.

[GoogleGuy]

Yah, you could also just put up an empty robots.txt file.

[AthlonInside]

Have you play starcraft?

The more SCVs or Drones or Probes you have the faster you mine the minerals and gas.

So the more crawler Google have, the faster they can mine their 'minerals'. :)