Msg#: 1917 posted 12:01 am on Jan 28, 2003 (gmt 0)
Recently found a bunch of hits from AV in my logs, which is nothing new. As usual, the IP addresses resolved to buildrack73.sv.av.com and the like. What's different, however, is that the hits show a referral URL, which is something I've never seen before when being crawled. Can anyone shed some light on this?
AltaVista appeared to be using an experimental crawler at the time which had limited support for referers and cookies, albeit referring page *within* my site. I emailed the contact addy and got no reply, which isn't a big surprise.
I don't think I've seen it since - but then again I'm not paying them :)
From my point of view I think referer was less surprising than the fact they seemed to be coping with a session cookie which was pretty impressive since no other commercial SE crawlers do this to my knowledge.