hi there, am looking for a comprehensive list of bot/spider IP addresses. I've been using [iplists.com...] but have recently found out that it doesn't list Alexa. Does anyone know of a comprehensive list that includes directories such as Alexa?
Hopefully this is the right forum for this post:)
Sanenet
2:01 pm on Jul 19, 2004 (gmt 0)
I don't think that Alexa has a bot (it's a browser object on users computers).
However, I did think that iplists.com was one of the best lists I found when I went through a recent "bot" phase :)
rchilakapati
2:25 pm on Jul 19, 2004 (gmt 0)
I believe Alexa does crawl sites. I've seen a lot of traffic from 209.237.238.**** and doing some research online, have found others with similiar issues. Has anyone else seen such a problem?
Thanks for the heads up! Are there other such crawlers out there that are not specified in iplists?
Sanenet
2:58 pm on Jul 19, 2004 (gmt 0)
ia_archiver also puts you into the Internet Archive - seems to be the main reason that Alexa keeps crawling the web. (That's how I knew the bot - didn't realize it still went through Alexa!)
