homepage Welcome to WebmasterWorld Guest from 54.211.217.197
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification

Search Engine Spider and User Agent Identification

Forum Charter, Library, Moderated by: Ocean10000 & incrediBILL Forum Options: Reset Last Read Pointers

PreModerated forum to id search engine spiders

 
new Subject Messages Started by Last Message
newpost post is pinned to top of list thread iconServer Farms - August 2014[9] ( 1 2 3 4 5 6 7 8 9 )
  Tracking and Reporting Data Center IP Ranges
266incrediBILL 12:22 am on Jan 24, 2015 (gmt 0)
newpost post is pinned to top of list thread iconDefault User Agents of Programming Libraries and Command Line Tools
  Resource page for common user agents
15incrediBILL 8:54 pm on Apr 29, 2008 (gmt 0)
newpost post is pinned to top of list thread iconQuick primer on identifying bot activity.
  And a how to guide to slow and stop scraping
21Ocean10000 12:48 pm on Apr 4, 2008 (gmt 0)
newpost thread iconDataline
  dtln.ru cloud servers
8keyplyr 11:52 am on Jan 28, 2015 (gmt 0)
newpost thread iconLooks like spiders from China?
  Receiving responsecode 500 errors from China visitors
2Anders 8:48 am on Jan 28, 2015 (gmt 0)
newpost thread iconAdvBot
 
9lucy24 9:32 pm on Jan 24, 2015 (gmt 0)
newpost thread iconMegaLine, KG
 
1keyplyr 9:04 am on Jan 23, 2015 (gmt 0)
newpost thread iconOdd User Agent
 
13wilderness 8:32 pm on Jan 22, 2015 (gmt 0)
newpost thread iconAutomating the server farm identification[2] ( 1 2 )
  an alternative approach
60trintragula 11:16 pm on Jan 20, 2015 (gmt 0)
newpost thread icon"Mozilla/5.0 ()" - Is this used by any valid browsers?
 
7physics 12:15 am on Jan 19, 2015 (gmt 0)
newpost thread iconAntiCrawler
  New bot crawler
4dstiles 10:39 pm on Jan 18, 2015 (gmt 0)
newpost thread iconFacebook app?
 
2lucy24 10:34 pm on Jan 14, 2015 (gmt 0)
newpost thread iconJiveSearchBot
 
1keyplyr 5:13 am on Jan 14, 2015 (gmt 0)
newpost thread iconNamed distributed crawler list
 
6trintragula 11:03 am on Jan 9, 2015 (gmt 0)
newpost thread iconHTTP header fields
  ACCEPT and ACCEPT-CHARSET
3dstiles 5:01 pm on Jan 8, 2015 (gmt 0)
newpost thread iconOpera Mini IP ranges
 
5Xpat 5:22 pm on Jan 7, 2015 (gmt 0)
newpost thread iconChinese exploiter out of Kansas City IP
 
10slipkid 6:34 am on Jan 6, 2015 (gmt 0)
newpost thread iconBad Bot Blackhole - Blocking Ranges
 
14SerpsGuy 1:21 am on Jan 1, 2015 (gmt 0)
newpost thread iconBaidu UA
 
5wilderness 10:05 am on Dec 31, 2014 (gmt 0)
newpost thread iconVisiStat
  Misbehaving San Jose, CA Brat blocking itself
3Hobbs 10:01 am on Dec 31, 2014 (gmt 0)
newpost thread iconGoogle-Fake Directories testing 404
 
23wilderness 5:10 am on Dec 18, 2014 (gmt 0)
newpost thread iconanother one for the profilers[2] ( 1 2 )
 
38lucy24 3:30 am on Dec 14, 2014 (gmt 0)
newpost thread iconDdos attack - what to do
 
21doc_z 12:37 pm on Dec 13, 2014 (gmt 0)
newpost thread iconWhy so many China bots?
 
13roshaoar 11:57 pm on Dec 8, 2014 (gmt 0)
newpost thread iconNerdyBot and the Google Cloud[2] ( 1 2 )
  Google range with strange referrer
54dupres01 11:40 am on Dec 5, 2014 (gmt 0)
Displaying Page 1 of 216 [ 1 ] 2 3 4 5 6 7 8 9 10 ... 216 Next >>
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved