homepage Welcome to WebmasterWorld Guest from 107.22.45.61
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification

Search Engine Spider and User Agent Identification

Forum Charter, Library, Moderated by: Ocean10000 & incrediBILL Forum Options: Reset Last Read Pointers

PreModerated forum to id search engine spiders

 
new Subject Messages Started by Last Message
newpost post is pinned to top of list thread iconServer Farms - April 2014[2] ( 1 2 )
  Tracking and Reporting Data Center IP Ranges
41incrediBILL 6:50 pm on Apr 23, 2014 (gmt 0)
newpost post is pinned to top of list thread iconDefault User Agents of Programming Libraries and Command Line Tools
  Resource page for common user agents
15incrediBILL 8:54 pm on Apr 29, 2008 (gmt 0)
newpost post is pinned to top of list thread iconQuick primer on identifying bot activity.
  And a how to guide to slow and stop scraping
21Ocean10000 12:48 pm on Apr 4, 2008 (gmt 0)
newpost thread iconBlock non-North American Traffic for Dummies Like Me[3] ( 1 2 3 )
  Reducing the size of your blocking list.
87webcentric 12:34 am on Apr 24, 2014 (gmt 0)
newpost thread iconSquider
  major hitter over the past ten days
1dstiles 8:42 pm on Apr 23, 2014 (gmt 0)
newpost thread iconu12Bot
 
1keyplyr 6:39 pm on Apr 21, 2014 (gmt 0)
newpost thread icon) )
 
14wilderness 4:30 am on Apr 20, 2014 (gmt 0)
newpost thread iconCrawling by Google
  How google crawling algo works
9hayden 7:03 am on Apr 18, 2014 (gmt 0)
newpost thread iconHow To Block Possible Click Bombing
 
20EmptyRoom 5:03 am on Apr 15, 2014 (gmt 0)
newpost thread iconCan the Googlebot read the blocking rules inside .htaccess?
 
5Andy500 8:55 pm on Apr 14, 2014 (gmt 0)
newpost thread iconCan Cloudfare stop content scrapers?
 
4Andy500 5:12 pm on Apr 14, 2014 (gmt 0)
newpost thread iconLevel 3
  How to handle?
9keyplyr 11:34 pm on Apr 11, 2014 (gmt 0)
newpost thread iconZeek-heartbleed-notifyer
 
1keyplyr 9:17 pm on Apr 10, 2014 (gmt 0)
newpost thread iconEnsnare web security tool
  Ensnare Attack Detection Tool Hopes to Frustrate Hackers
1dstiles 8:11 pm on Apr 10, 2014 (gmt 0)
newpost thread iconShould I block msnbot-media
 
5shamrock 8:26 pm on Apr 9, 2014 (gmt 0)
newpost thread iconPrevious Server Farms Threads
 
4incrediBILL 4:46 pm on Apr 9, 2014 (gmt 0)
newpost thread iconShould blank user agents be blocked or not?
 
5aristotle 7:47 pm on Apr 7, 2014 (gmt 0)
newpost thread iconServer Farms - Feb. 2014[4] ( 1 2 3 4 )
  Ongoing Hosting Data Center Discussion
109incrediBILL 5:43 pm on Apr 7, 2014 (gmt 0)
newpost thread iconNerdyBot
  Google range with strange referrer
5dupres01 5:35 pm on Apr 7, 2014 (gmt 0)
newpost thread iconIs this googlebot or not?
  Seems this in GBot's Range but doesn't Validate
14webcentric 2:36 pm on Apr 7, 2014 (gmt 0)
newpost thread iconupdown tester
  Digital Ocean ip ranges
1Hobbs 10:55 am on Apr 7, 2014 (gmt 0)
newpost thread iconHow to block googleusercontent.com
  blocking google and googleusercontent in htaccess
8rogerroger 5:33 am on Apr 7, 2014 (gmt 0)
newpost thread iconAmazon AWS Hosts Bad Bots[2] ( 1 2 )
  Continuation Thread
44incrediBILL 9:47 pm on Apr 5, 2014 (gmt 0)
newpost thread iconBots Crawling Pages Blocked by Robots.txt
 
7incrediBILL 6:52 am on Apr 5, 2014 (gmt 0)
newpost thread iconIs there a list of user-agents to block as of today?
  I'm launching a new website soon and want to avoid scrapers
6shamrock 6:06 am on Apr 5, 2014 (gmt 0)
Displaying Page 1 of 213 [ 1 ] 2 3 4 5 6 7 8 9 10 ... 213 Next >>
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved