homepage Welcome to WebmasterWorld Guest from 54.145.183.126
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification

Search Engine Spider and User Agent Identification

Forum Charter, Library, Moderated by: Ocean10000 & incrediBILL Forum Options: Reset Last Read Pointers

PreModerated forum to id search engine spiders

 
new Subject Messages Started by Last Message
newpost thread iconEuripBot
 
3keyplyr 8:46 pm on Jan 14, 2012 (gmt 0)
newpost thread iconMS Crawler Hiding as a Browser
 
14incrediBILL 4:19 am on Jan 11, 2012 (gmt 0)
newpost thread iconIs this a valid user agent? and how do I block it?
  Is this a valid user agent? and how do I block it?
12spiritualseo 4:59 am on Jan 10, 2012 (gmt 0)
newpost thread iconNook
 
2Pfui 7:22 pm on Jan 8, 2012 (gmt 0)
newpost thread iconServers using open proxies
 
4dstiles 3:21 pm on Jan 7, 2012 (gmt 0)
newpost thread iconboia.org
  tools request, ignore robots.txt
2Pfui 7:06 am on Jan 7, 2012 (gmt 0)
newpost thread iconGooglebot-richsnippets
 
2Pfui 2:37 am on Jan 7, 2012 (gmt 0)
newpost thread iconMFE expand
 
3Pfui 12:44 am on Jan 7, 2012 (gmt 0)
newpost thread iconVocus
  Hocus Pocus
2Pfui 11:17 pm on Jan 6, 2012 (gmt 0)
newpost thread iconaiHitBot
 
1Pfui 8:46 pm on Jan 6, 2012 (gmt 0)
newpost thread iconQuora Link Preview
 
1Pfui 7:30 pm on Jan 6, 2012 (gmt 0)
newpost thread iconKindle & Amazon IPs?
 
28keyplyr 8:45 pm on Jan 5, 2012 (gmt 0)
newpost thread iconMozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
 
9Pfui 8:41 pm on Jan 5, 2012 (gmt 0)
newpost thread iconAutomatically detect crawlers / bots
 
13Globetrotter 3:03 pm on Jan 5, 2012 (gmt 0)
newpost thread iconSemrushbot/0.9
 
5tangor 11:12 pm on Jan 4, 2012 (gmt 0)
newpost thread iconyolinkBot
 
4keyplyr 7:06 pm on Jan 3, 2012 (gmt 0)
newpost thread iconMSNBot has become a constant Fast-Scraper[2] ( 1 2 )
  7 IPs crawling at max 12 pages / sec - this is out of order
42AlexK 8:28 pm on Jan 1, 2012 (gmt 0)
newpost thread iconGoogleDocs
 
4Pfui 3:09 am on Dec 31, 2011 (gmt 0)
newpost thread iconDDDD DDDD human?
 
8lucy24 2:39 am on Dec 29, 2011 (gmt 0)
newpost thread iconmidphase.com
 
1keyplyr 1:36 pm on Dec 27, 2011 (gmt 0)
newpost thread iconPacket One
 
2keyplyr 5:07 pm on Dec 24, 2011 (gmt 0)
newpost thread iconWHOIS XML RESTful APIs!
 
1incrediBILL 2:42 am on Dec 24, 2011 (gmt 0)
newpost thread iconAwstats Scrapers / Injectors
 
19dstiles 11:59 pm on Dec 19, 2011 (gmt 0)
newpost thread iconNew scraper? portalimage.org
 
9brokaddr 11:27 am on Dec 17, 2011 (gmt 0)
newpost thread iconCrystalSemanticsBot
  advertising recon
1keyplyr 9:49 pm on Dec 16, 2011 (gmt 0)
Displaying Page 27 of 216<< Back 1 ... 17 18 19 20 21 22 23 24 25 26 [ 27 ] 28 29 30 31 32 33 34 35 36 ... 216 Next >>
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved