Forum Moderators: phranque
Googlebot - 2783+48
GigaBot - 155+5
Yahoo Slurp - 50+33
Unknown robot (identified by 'crawl') - 64+11
MSNBot-media - 30+8
MSNBot - 5+8
I have googlebot pretty much around the clock on my site but what the matter with the other spiders? Any ideas?
My method:
For each GET *.html entry in my access log, search for bots first (case insensitive):
Google: googlebot
MSN: msnbot
Yahoo: slurp
Once these entries are removed, search for a user hit from the engine:
Google: .google.
MSN: msn.co OR search.live
Yahoo: search.yahoo
Is this a correct method, or am I missing user hits from MSN or Yahoo?