homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Gold Sponsor 2015!
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Web Accelerators

5+ Year Member

Msg#: 3252128 posted 6:40 am on Feb 14, 2007 (gmt 0)

Can anyone give me any tips on how to tell the differance between a bot or if it's just a browser with a web accelerator?

What I'm seeing is a regular UA but it doesn't always get images. It starts out normal but ends with no images. It does this at a "human rate" and doesn't take very many pages. I see this from different IPs and they always go to different parts of the site, just like normal.

Another normal UA I see seems to be multi threaded in that it gets 2 pages in the same second. Sometimes the pages are the same page called twice and other times it calls default and the second page called is different, but at the same second.

Clear as mud I know.

Are these web accelerators? Or bots or browsers with the images disabled or am I just looking at log files too much?




WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3252128 posted 1:44 am on Feb 15, 2007 (gmt 0)

most web accelerators (or similar tools), although perhaps NOT identified with an appropiate UA, will come from a reputable IP range of either a search engine utilizing some type or tool or a software that is exclusively for acceleration.

There's some old threads on a few of these


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved