Forum Moderators: DixonJones
I set up a new site late November and am trying to interpret my stats for December 2004. Basically I'm wondering, for a new site, are these robot visits reasonable or do I need to be getting more attention from spiders? Let me know your response to these:
Googlebot 5611
Unknown robot (identified by 'crawl') 4255
Wget 612
Inktomi Slurp 542
BaiDuSpider 294
psBot 101
Unknown robot (identified by 'robot') 38
larbin 15
LinkWalker 14
Others 28
Also can anyone tell me what robot "Unknown robot (identified by 'crawl')" is? Considering it's second only to Googlebot it's rather unusual for it to be "unknown"!
Thx and enjoy 2005 :)
Jeremy
A lot depends on how many pages your site has ... and how many visits are involved, in combination with how many others' pages are linked to yours. If Google has made 5,000 separate visits for 5,#*$! pages, and you have a 5,000 page site, Google isn't getting very much of your site, but it's coming a lot. Slurp, as usual, is kinda pitiful by comparison (come on, Yahoo, get it together!), but it could be getting your whole site, which makes it adequate.
If you can get a report for each of the big 'bots and look at the page list, you'll have a better idea of whether you're getting the right amount of attention (i.e. a thorough crawl).
Conspicuously missing from your list is MSNBot which has been outgunning even Google on my sites, recently. See if you can add it somehow.
I wouldn't be unhappy with that amount of SE business in that period of time, but the real measure is going to be how you show up in results.
Happy New Year!
Jeremy
In AwStats (cPanel) the unknowns are usually only unknown to the AwStat Statistics software until the software is updated and when updated the unknown bots would be listed as *known by their name
When the new MSN bot came out
msnbot/0.3_(_http://search.msn.com/msnbot.htm)
it was listed as unknown in the 9.9.8-RELEASE 119 of cPanel (AwStats)
Unknown identified by* - Could be a new MSN bot or even a rouge link checker
