Forum Moderators: bakedjake

Message Too Old, No Replies

AOL is driving me crazy!

         

flexsez

5:44 am on Nov 9, 2002 (gmt 0)

10+ Year Member



Can anyone tell me why the AOL spider is constantly spidering my site? That bleepin spider comes everyday and searches about 20 pages (and my sites only 35 pages).

And the funny thing is, I only get about 5 visits from AOL each day.

It's starting to annoy me.

Anyone have a similar experience?

jdMorgan

1:38 am on Nov 10, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



flexsez,

AOL serves as many requests as they can from their caching system. It is likely that this is what you are seeing. Because of the heavy caching they employ, you will see very few requests from AOL users unless you take steps to force AOL to load the pages/images/etc. from your server rather than serving them from their cache. Also for this reason, your AOL traffic is likely to be much higher than you think it is.

Do a site search for "cache-control headers" here on WebmasterWorld - there is a recent thread on the subject.

A common technique is to "mark" one of your common-to-all-pages images, preferably a small one, as non-cacheable. That way, you will see all AOL-user requests for this one page element, but go ahead and let them load most pages and pages elements from the AOL cache to improve performance and reduce your server load. If you need to collect referer info, you will have to force AOL to not cache your home page or most-popular entry pages, rather than using the small-image trick.

HTH,
Jim

msr986

1:44 am on Nov 10, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> If you need to collect referer info, you will have to force AOL to not cache your home page or most-popular entry pages, rather than using the small-image trick.

Is there any down side to doing this? (Other than increased server traffic).

Are any other major ISP's doing this?

jdMorgan

2:03 am on Nov 10, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



msr986,

The downside is slower page-loading for AOL'ers and more load on your server. If you have a valid reason to stop them from caching then do it. Even if you have pages that change often, it's better to simply provide a last-modified header when you serve a page, rather than stopping all caching. If you don't need to stop them from caching, then don't - reducing net traffic is a good thing.

Major ISP's doing it? I don't know really - AOL is the only big one I know of. A small ISP in my area uses a cache though, and I have tweaked my server headers to work well with them and with AOL.

Here's a decent cacheability checker [ircache.net] - see the main page for tutorials.

Related thread [webmasterworld.com].

Jim