Forum Moderators: DixonJones

Message Too Old, No Replies

How do I know which search engine is visiting me?

         

ihajjee

12:03 pm on Mar 4, 2004 (gmt 0)

10+ Year Member



Hi,

I constantly get visits to robot.txt (though I don't have one), and I also see stuff like:

Googlebot/2.1
QuepasaCreep
msnbot/0.11

is googlebot the search engine for Google?
msnbot for msn?

how do I know when yahoo, altavista, etc visit me?

sem4u

12:10 pm on Mar 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Googlebot is the spider for Google, MSNbot is the spider for MSN's new index (they are building one).

Yahoo's crawler is Yahoo! Slurp, Altavista's is Scooter.

Beach_lady

11:47 pm on Mar 4, 2004 (gmt 0)

10+ Year Member



I also have a question about this. I had goolge search go through my site but it was not the same as the google search that shows the search terms. This just said google search. Is this a spider? Also what is DigExt, Hotbar, & Gecko? I thought somewhere on these threads I read that Hotbar and FunWebProducts were nothing we wanted visiting. If this is so what do I add to my Robot.txt to prevent them from coming. Also Yahoo Slurp and Scooter come often but I have not seen them request the robot.txt. They just proceed.

knoir

6:21 pm on Mar 9, 2004 (gmt 0)

10+ Year Member



for log analyzing, i use weblog expert. It is a great program to analyze your log files. You can read more about it here: [snip]

I hope this helps. It breaks down your site log into many categories. Entry pages, exit pages, countries, search engines, keywords, etc.

Enjoy

[edited by: pageoneresults at 8:37 pm (utc) on Mar. 9, 2004]
[edit reason] Removed URI - Please refer to TOS, thanks. [/edit]