Forum Moderators: DixonJones
For example. I modified the robots.pm file to include:
'sproose\scrawler',
'sproose',
'sproose\scrawler','<a href="http://www.sproose.com/bot.html" title="Bot home page [new window]" target="_blank">sproose crawler</a>',
'sproose','<a href="http://www.sproose.com/bot.html" title="Bot home page [new window]" target="_blank">sproose</a>',
After a few days, sproose bot does not show up in the full list of robots under awstats. Despite the fact that I can see hits from the Sproose bot in my log files and awstats has updated:
domain.com.log:38.100.225.6 - - [14/Oct/2006:07:14:53 -0700] "GET /robots.txt HTTP/1.0" 200 220 "-" "sproose/1.0beta (sproose bot; [sproose.com...] crawler@sproose.com)"
domain.com.log:38.100.225.6 - - [14/Oct/2006:07:14:53 -0700] "GET /blog/ HTTP/1.0" 302 1797 "-" "sproose/1.0beta (sproose bot; [sproose.com...] crawler@sproose.com)"
domain.com.log:38.100.225.6 - - [14/Oct/2006:07:14:56 -0700] "GET /blog/formic.php HTTP/1.0" 302 0 "-" "sproose/1.0beta (sproose bot; [sproose.com...] crawler@sproose.com)"
I have checked and double checked the robots.pm file and everything looks fine. It's picking up other bots in there without a problem. I even downloaded a pre-made robots.pm file that includes support for detecting the Sproose robot from [antezeta.com...] -- waited a couple of days, and am still not seeing Sproose in the full robots list
I also searched awstats data files for the IP address for Sproose bot that is showing up in my logs, and its not being logged in there anywhere. This leads me to believe that Awstats is reconizing the hit as a bot and is not counting it. But does not explain why its not showing up in the robots list.
Any ideas?
Cheers!
