how do I know which search engine is visitng me?

Forum Moderators: open

Message Too Old, No Replies

how do I know which search engine is visitng me?

ihajjee

12:03 pm on Mar 4, 2004 (gmt 0)

Hi,

I constantly get visits to robot.txt (though I don't have one), and I also see stuff like:

Googlebot/2.1
QuepasaCreep
msnbot/0.11

is googlebot the search engine for Google?
msnbot for msn?

how do I know when yahoo, altavista, etc visit me?

sem4u

12:10 pm on Mar 4, 2004 (gmt 0)

Googlebot is the spider for Google, MSNbot is the spider for MSN's new index (they are building one).

Yahoo's crawler is Yahoo! Slurp, Altavista's is Scooter.

Beach_lady

11:47 pm on Mar 4, 2004 (gmt 0)

I also have a question about this. I had goolge search go through my site but it was not the same as the google search that shows the search terms. This just said google search. Is this a spider? Also what is DigExt, Hotbar, & Gecko? I thought somewhere on these threads I read that Hotbar and FunWebProducts were nothing we wanted visiting. If this is so what do I add to my Robot.txt to prevent them from coming. Also Yahoo Slurp and Scooter come often but I have not seen them request the robot.txt. They just proceed.