homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
Crawling with referrals: Is it possible?
It appears to be AV, but the hits look like referrals

 12:01 am on Jan 28, 2003 (gmt 0)

Recently found a bunch of hits from AV in my logs, which is nothing new. As usual, the IP addresses resolved to buildrack73.sv.av.com and the like. What's different, however, is that the hits show a referral URL, which is something I've never seen before when being crawled. Can anyone shed some light on this?



 8:47 am on Jan 29, 2003 (gmt 0)

Have you checked the IP address is AltaVista and not just someone playing fake UA games?

I've posted something similar about this a month or so ago.


AltaVista appeared to be using an experimental crawler at the time which had limited support for referers and cookies, albeit referring page *within* my site. I emailed the contact addy and got no reply, which isn't a big surprise.

I don't think I've seen it since - but then again I'm not paying them :)

From my point of view I think referer was less surprising than the fact they seemed to be coping with a session cookie which was pretty impressive since no other commercial SE crawlers do this to my knowledge.



 9:09 am on Jan 29, 2003 (gmt 0)

AV has been testing a new bot with some new functions to better simulate human visitors. I don't know what the specifics are so, this wouldn't surprise me.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved