homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

AOL's Spiders?
They have been visiting my site for 1 week.

10+ Year Member

Msg#: 107 posted 8:59 pm on Aug 14, 2000 (gmt 0)

I just wondering what are they doing in my site.
They kept visiting my site for about 1 week but I don't know what they did. Should I do something to raise my rank?



WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 107 posted 6:56 am on Aug 16, 2000 (gmt 0)

An age old question suitable for a FAQ file.

AOL does not really have spiders. What they have a proxy agents. The way it works is they download the page for users.

Let's say you are using AOL for an ISP service. You want to view this very page. You click on on your WebmasterWorld bookmark. AOL then sends out a 'spider' or proxy agent to download the page. AOL then serves the user that page. The next time somebody else wants the page, aol just gives them the copy out of it's cache and not connection is made.

In order to serve all the possible combinations of a site, AOL must change the user agent string of its proxy spiders/agents. You can sometimes get 10-20 hits from AOL 'spiders' (they have spider in their domain name) in rapid fire succession.

Try the AOL Webmaster FAQ [webmaster.aol.com] for more information (IE only I think). Look at the Proxy and Cache items on the left menu.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved