Referer not an existing page?

How can that be?

[Sinner_G]

I am looking through my log files (from IIS server). I was puzzled to see that pages were looked for which don't exist any more since a redesign more than a year ago. So I checked what the refering URL was. And it was another page from my site, which also doesn't exist any more since said redesign.

How is that possible? Could it be from some cache that hasn't been updated for that long?

[sugarkane]

A strange one... do you have the user agent for those requests?

[Sinner_G]

Yes, but that makes the whole thing even more weird. I get things like Mozilla/4.0+(compatible;+MSIE+5.0;+Windows+98;+DigExt), which I'm told is the "make available offline" tool of IE, but also spiders (e.g. the AskJeeves bot) and what seems to be regular users.

I just don't get it.

[Dreamquick]

Have you considered that for some reason or other they could just be faking the referrer? I'm surprised that an SE's bot would leave a referrer at all, the majority of the ones I've seen just leave "referer" blank.

In my experience I see something close and possibly related - certain SE's not bothering to remove pages which return 404 for a very long time from their crawl DB, but I don't think I've ever seen one of their *crawlers* return a referral url!

If you haven't done it already check to see if the "crawler" ip address is what they are supposed to be, if it's not then you have an imposter posing as the crawler but if matches then potentially it might just be an employee checking out something...

Lastly your "digext" browser could be an independant bot with a fake UA running off a *really* old set of results or just exactly what you suggested - a user running offline browsing, maybe they were trying to update a really old version of their page (do they re-use the orignal request, and therefore the original referral url?).

- Tony