How do I know which search engine is visiting me?

[ihajjee]

Hi,

I constantly get visits to robot.txt (though I don't have one), and I also see stuff like:

Googlebot/2.1
QuepasaCreep
msnbot/0.11

is googlebot the search engine for Google?
msnbot for msn?

how do I know when yahoo, altavista, etc visit me?

[sem4u]

Googlebot is the spider for Google, MSNbot is the spider for MSN's new index (they are building one).

Yahoo's crawler is Yahoo! Slurp, Altavista's is Scooter.

[Beach_lady]

I also have a question about this. I had goolge search go through my site but it was not the same as the google search that shows the search terms. This just said google search. Is this a spider? Also what is DigExt, Hotbar, & Gecko? I thought somewhere on these threads I read that Hotbar and FunWebProducts were nothing we wanted visiting. If this is so what do I add to my Robot.txt to prevent them from coming. Also Yahoo Slurp and Scooter come often but I have not seen them request the robot.txt. They just proceed.

[knoir]

for log analyzing, i use weblog expert. It is a great program to analyze your log files. You can read more about it here: [snip]

I hope this helps. It breaks down your site log into many categories. Entry pages, exit pages, countries, search engines, keywords, etc.

Enjoy

[edited by: pageoneresults at 8:37 pm (utc) on Mar. 9, 2004]
[edit reason] Removed URI - Please refer to TOS, thanks. [/edit]