Welcome to WebmasterWorld Guest from 54.145.208.64

Forum Moderators: open

Message Too Old, No Replies

Crawling with referrals: Is it possible?

It appears to be AV, but the hits look like referrals

   
12:01 am on Jan 28, 2003 (gmt 0)

10+ Year Member



Recently found a bunch of hits from AV in my logs, which is nothing new. As usual, the IP addresses resolved to buildrack73.sv.av.com and the like. What's different, however, is that the hits show a referral URL, which is something I've never seen before when being crawled. Can anyone shed some light on this?
8:47 am on Jan 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Have you checked the IP address is AltaVista and not just someone playing fake UA games?

I've posted something similar about this a month or so ago.

[webmasterworld.com...]

AltaVista appeared to be using an experimental crawler at the time which had limited support for referers and cookies, albeit referring page *within* my site. I emailed the contact addy and got no reply, which isn't a big surprise.

I don't think I've seen it since - but then again I'm not paying them :)

From my point of view I think referer was less surprising than the fact they seemed to be coping with a session cookie which was pretty impressive since no other commercial SE crawlers do this to my knowledge.

-Tony

9:09 am on Jan 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



AV has been testing a new bot with some new functions to better simulate human visitors. I don't know what the specifics are so, this wouldn't surprise me.

Chris