AOL Crawling?

What have you seen?

[seth_wilde]

I found this little bugger(spider-wd014.proxy.aol.com) following links on a site (it actually requested a chat with me via a human click link). Do you think this is the source of the "mysterious" results from AOL?

[JamesR]

>it actually requested a chat with me via a human click link). Do you think this is the source of the "mysterious" results from AOL?

You should ask it ;)

[littleman]

Hmm, you know at first glance I would dismiss it as a proxy/caching spider. But the crawling is interesting - some time ago Brett was wondering how some pages
that was intended to be for aol users only got into inktomi.

Is this site in ODP?

[NFFC]

Think it's just a host for AOL users, they do like their anime [excelsis.com]though!

[seth_wilde]

"You should ask it" *l* I tried but it wouldn't talk to me :)

"Think it's just a host for AOL users" Thanks NFFC, that's kind of an odd one for AOL. I wonder why I don't see this in the logs more?

[seth_wilde]

The AOL mystery remains unsolved :(

[NFFC]

Aol proxy rant here [dougm.com].

[tedster]

AOL is a very sharp outfit, and I imagine that hitting a page from 7 proxies at once has a real enough purpose from their perspecative. But they sure don't care whether our server logs make any kind of sense.

I have a client who, yesterday, asked me for the eighth or ninth time the question: "How many people visited my site today?" They looked at me in total frustration as I once again explained some of the reasons why there is no straightforward answer for this apparently straightforward question. Of course I can't lay it all on AOL, but it's fun to try.

[pete]

Hi All, I also thought that this was an AOL spider when I first took a look at it crawling our site. We see them regularly and they are not spiders but people. So, go along with NFFC regarding AOL host.

First picked this up when one of our consultants were prompted for a chat also via humanclick. But in our case, the spider talked back and asked for a safari trip :)

[Brett_Tabke]

Aol Joe User Clicks Link A, which is read and processed by the proxy server on Aol. If the proxy cache has that link in its cache, it serves the page right back to the user. If it doesn't have it in the cache, it sends out a spider to grab the page, cache it, and finally send it back to the user. Joe, happened to click on your chat link that wasn't cached.

[oilman]

>>hitting a page from 7 proxies at once

Yeah - that drives me nuts. We run live stats here and half our marketing department logs on every morning to see how many active sessions we have going and how our ad returns are etc. Every couple days someone says 'Hey look we have 50 active sessions! Cool!' and I have to go over and point out that 10 of them are AOL proxies and explain it all over again.