Forum Moderators: phranque
Inktomi Slurp
Googlebot
MSNBot
Unknown robot (identified by 'crawl')
Unknown robot (identified by hit on 'robots.txt')
Uknown robot (identified by 'spider')
AskJeeves
Alexa (IA Archiver)
Unknown robot (identified by 'robot')
I've submitted my sites properly many times without over submitting & was wondering if there are any robots I'm missing out on which would maybe do my site any favours, if so how can I reach them?
It doesn't really detect spiders real well unless you personally develope the libraries. I'm reworking my version before I go public again (security hole) but yeah, if you play with awstats and mod the libraries a little and look in to things you can really see whats going on. :-)