Forum Moderators: phranque

Message Too Old, No Replies

Crawling behavior of other spiders

         

eRAZOR

1:10 pm on Nov 24, 2007 (gmt 0)

10+ Year Member



The following tables shows the hits to my site (it's a very young site) for the various spiders (the number after the plus sign is the number of requests for robots.txt):

Googlebot - 2783+48
GigaBot - 155+5
Yahoo Slurp - 50+33
Unknown robot (identified by 'crawl') - 64+11
MSNBot-media - 30+8
MSNBot - 5+8

I have googlebot pretty much around the clock on my site but what the matter with the other spiders? Any ideas?

turkeyproperty

11:52 pm on Nov 26, 2007 (gmt 0)

10+ Year Member



recently my site ranked #1 in yahoo for a strong word "turkey
property" and #1 in google,

for the last few days however my rankings at yahoo have vanished
into the blue, no sign of main page.

also as mentioned crawlings have reduced majorly...

but only yahoo. whatz happen'n anyone?

newbie6

12:11 am on Nov 27, 2007 (gmt 0)

10+ Year Member



I have the opposite experience with a many years old site, most pages have been unchanged for over a year - Google gives me 10k user hits for only 200 bot requests, while Yahoo Slurp hits me 3k/month for only 1k user views, and MSNBot takes the cake with 1500 bot hits bu only 200 user views.

My method:
For each GET *.html entry in my access log, search for bots first (case insensitive):
Google: googlebot
MSN: msnbot
Yahoo: slurp
Once these entries are removed, search for a user hit from the engine:
Google: .google.
MSN: msn.co OR search.live
Yahoo: search.yahoo

Is this a correct method, or am I missing user hits from MSN or Yahoo?