Forum Moderators: open
What is strange is this search query at AOL resulted in a hit at my site where the html page was requested and returned a server 200 code even though I did not load the page. The other files embedded in the page including menu, css, background, etc. were not requested. The log shows IP and U/A, but no referrer. Note that I did not click on the link to my site in the SERPs. The visitor logged was definitely me as the IP matches.
Is this a common occurence? I have considered visitors that only download the html file to be some type of stealth bots. Given the suspicious activity, I have been banning such IPs with .htaccess. What is a better way to determine the stealth bots from real visitors?
Anyone ever see this before?
Ted
AOL is.... different. They use a heavy caching system, and their subscribers connect through AOL proxies - I'm sure you've seen AOLers changing IP addresses for each HTTP request to your site.
I suspect that AOL search was "pre-loading" their cache in case you clicked the link in the SERP. This is just supposition on my part, but it wouldn't surprise me if it were true.
BTW, which AOL search were you using when you followed the log entry - the "Connect Anywhere" version, or something else? What was the rank of your site in the SERP? I may have to go try this - it's interesting.
Thanks,
Jim
Yes, I'm aware of AOL using lots of different IPs with each visitor. This throws a new spin on blocking troublemakers.
I suppose it is AOL Anywhere. I don't subscribe to AOL, only Road Runner. I wonder if we have any AOL subscribers on this board? ... doubt it ... ;)
Try:
[aolsearch.aol.com...]
Replace www with your favorite serach term
My rank on that SERP was #1. Let me know what you uncover.
Ted
OK, I can partially confirm this phenomenon, and expand on it a little...
I did a search that resulted in one of my pages being listed as #1, and a second one listed as #2 as what would be an "indented" result if I had done the search on Google directly (it includes the "see more results from this site" link).
In my logs, I see requests with my IP address, my browser's user-agent string, and the AOL search URL as the referrer for both pages. AOL also requested all of the resources included on those two pages - images, external JavaScripts, etc.
Notably, included resources which are common to both pages were only fetched once, so this is an intelligent agent at work, here.
Most responses were 200-OK status, but some of the images and the second-listed html page show a 304-Not Modified response, indicating that AOL did a conditional GET, and already has these resources cached.
My pages which appeared on page 2 of the SERPs and later were not requested by AOL.
I don't want to over-analyze it, but there's some more data - AOL Search is doing some sore of "prefetching" on pages listed on the first page of the SERPs.
Jim
Occassionally I get refer links from AOL which have something added to the referral regarding either 7 or 8 bit something.
If I directly cut and paste these full referrals the result is that my browser crashes. I'm not sure whever it's because of the difference in bit browser compatabilities or the fact that I have all scriping and such disabled?
Don
A clarification:
> AOL also requested all of the resources included on those two pages - images, external JavaScripts, etc.
These were requested with the name of the first-listed html page as the referrer. The two pages themselves were requested with the AOL search URL as the referrer.
Don,
I've never seen a referrer like that - with the 7-/8-bit thing. Post if you see another one.
Jim
I've never seen a referrer like that - with the 7-/8-bit thing. Post if you see another one.
[google.com...]
