Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Crawling with referrals: Is it possible?

It appears to be AV, but the hits look like referrals



12:01 am on Jan 28, 2003 (gmt 0)

10+ Year Member

Recently found a bunch of hits from AV in my logs, which is nothing new. As usual, the IP addresses resolved to buildrack73.sv.av.com and the like. What's different, however, is that the hits show a referral URL, which is something I've never seen before when being crawled. Can anyone shed some light on this?


8:47 am on Jan 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Have you checked the IP address is AltaVista and not just someone playing fake UA games?

I've posted something similar about this a month or so ago.


AltaVista appeared to be using an experimental crawler at the time which had limited support for referers and cookies, albeit referring page *within* my site. I emailed the contact addy and got no reply, which isn't a big surprise.

I don't think I've seen it since - but then again I'm not paying them :)

From my point of view I think referer was less surprising than the fact they seemed to be coping with a session cookie which was pretty impressive since no other commercial SE crawlers do this to my knowledge.



9:09 am on Jan 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

AV has been testing a new bot with some new functions to better simulate human visitors. I don't know what the specifics are so, this wouldn't surprise me.



Featured Threads

Hot Threads This Week

Hot Threads This Month