Can someone please tell me what Alexa's spider is called and how it is recorded in your logs.
Thanks.
PS If you know the name of AOL's as well that would be useful.
bcolflesh
3:59 pm on Sep 23, 2003 (gmt 0)
pages.alexa.com/help/webmasters/
sidyadav
12:44 am on Sep 24, 2003 (gmt 0)
ia_archiver , it doesn't follow robots.txt , it doesn't follow links, I don't know how it hits certain pages of my site when it is not linked to any other pages.