homepage Welcome to WebmasterWorld Guest from 54.204.90.135
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification

Search Engine Spider and User Agent Identification

Forum Charter, Library, Moderated by: Ocean10000 & incrediBILL Forum Options: Reset Last Read Pointers

PreModerated forum to id search engine spiders

 
new Subject Messages Started by Last Message
newpost post is pinned to top of list thread iconServer Farms - August 2014[3] ( 1 2 3 )
  Tracking and Reporting Data Center IP Ranges
61incrediBILL 9:07 pm on Sep 20, 2014 (gmt 0)
newpost post is pinned to top of list thread iconDefault User Agents of Programming Libraries and Command Line Tools
  Resource page for common user agents
15incrediBILL 8:54 pm on Apr 29, 2008 (gmt 0)
newpost post is pinned to top of list thread iconQuick primer on identifying bot activity.
  And a how to guide to slow and stop scraping
21Ocean10000 12:48 pm on Apr 4, 2008 (gmt 0)
newpost thread iconDon't understand this log entry
 
14aristotle 8:42 pm on Sep 20, 2014 (gmt 0)
newpost thread iconNerdyBot and the Google Cloud
  Google range with strange referrer
24dupres01 2:40 am on Sep 3, 2014 (gmt 0)
newpost thread iconContextAdbot IP updates
 
6lucy24 1:31 am on Sep 3, 2014 (gmt 0)
newpost thread iconGTT bots
  You may want to fine tune your blocks: I don't
5Angonasec 12:22 pm on Aug 27, 2014 (gmt 0)
newpost thread iconstrange googlebot referer
  googlebot referral from odd site
9dupres01 7:06 am on Aug 27, 2014 (gmt 0)
newpost thread iconxovibot
 
7lucy24 7:05 pm on Aug 25, 2014 (gmt 0)
newpost thread iconVoltron
  a new-ish 80legs crawler
1dstiles 6:28 pm on Aug 24, 2014 (gmt 0)
newpost thread iconodd apple requests
  strange requests coming from apple ramge
6dupres01 6:20 pm on Aug 24, 2014 (gmt 0)
newpost thread iconAmazon AWS Hosts Bad Bots[2] ( 1 2 )
  Continuation Thread
59incrediBILL 1:27 am on Aug 20, 2014 (gmt 0)
newpost thread iconHubSpot Webcrawler
  (more AWS spawn)
1Pfui 11:14 pm on Aug 19, 2014 (gmt 0)
newpost thread iconIPv6 enabled Bots / Server Farm
 
8Ocean10000 6:24 pm on Aug 18, 2014 (gmt 0)
newpost thread iconGooglebot and iPhone
 
30wilderness 1:41 pm on Aug 17, 2014 (gmt 0)
newpost thread iconManticore
  (more AWS spawn)
5Pfui 6:06 pm on Aug 16, 2014 (gmt 0)
newpost thread iconEnterprise Search Appliances for Scrapers?
  Thunderstone
20not2easy 9:56 am on Aug 14, 2014 (gmt 0)
newpost thread iconstremorbot
 
4wilderness 1:38 am on Aug 14, 2014 (gmt 0)
newpost thread iconapache pb2.gif[2] ( 1 2 )
  why do bots ask for this
44dupres01 4:46 am on Aug 12, 2014 (gmt 0)
newpost thread iconAug 8 Amazon UPDATED public IPs
  Aws ip
4miscbyproduct 4:00 pm on Aug 11, 2014 (gmt 0)
newpost thread iconA6 on its Knees
  They are taking notice
9Angonasec 12:06 pm on Aug 11, 2014 (gmt 0)
newpost thread iconsub-semalt
 
10wilderness 7:05 pm on Aug 10, 2014 (gmt 0)
newpost thread iconPHP Vulnerability
 
1wilderness 11:23 am on Aug 9, 2014 (gmt 0)
newpost thread iconMicrosoft Insitesbot
 
6Hobbs 7:13 pm on Aug 7, 2014 (gmt 0)
newpost thread iconBrandProtect
 
5lucy24 12:44 am on Aug 2, 2014 (gmt 0)
Displaying Page 1 of 215 [ 1 ] 2 3 4 5 6 7 8 9 10 ... 215 Next >>
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved