Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

AOL's Spiders?

They have been visiting my site for 1 week.



8:59 pm on Aug 14, 2000 (gmt 0)

Inactive Member
Account Expired


I just wondering what are they doing in my site.
They kept visiting my site for about 1 week but I don't know what they did. Should I do something to raise my rank?
6:56 am on Aug 16, 2000 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 11

An age old question suitable for a FAQ file.

AOL does not really have spiders. What they have a proxy agents. The way it works is they download the page for users.

Let's say you are using AOL for an ISP service. You want to view this very page. You click on on your WebmasterWorld bookmark. AOL then sends out a 'spider' or proxy agent to download the page. AOL then serves the user that page. The next time somebody else wants the page, aol just gives them the copy out of it's cache and not connection is made.

In order to serve all the possible combinations of a site, AOL must change the user agent string of its proxy spiders/agents. You can sometimes get 10-20 hits from AOL 'spiders' (they have spider in their domain name) in rapid fire succession.

Try the AOL Webmaster FAQ [webmaster.aol.com] for more information (IE only I think). Look at the Proxy and Cache items on the left menu.