homepage Welcome to WebmasterWorld Guest from 54.205.254.108
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification

Search Engine Spider and User Agent Identification

Forum Charter, Library, Moderated by: Ocean10000 & incrediBILL Forum Options: Reset Last Read Pointers

PreModerated forum to id search engine spiders

 
new Subject Messages Started by Last Message
newpost thread iconPromptCloud bot
  Web Extraction, crawling and scraping service
4AnnieRogers 8:19 pm on Aug 1, 2013 (gmt 0)
newpost thread iconEvernote
 
2wilderness 3:23 am on Jul 30, 2013 (gmt 0)
newpost thread iconmozilla
  position of "Mozilla" in UA string
4lucy24 6:50 pm on Jul 24, 2013 (gmt 0)
newpost thread iconBaidu Hong Kong
 
2lucy24 6:10 pm on Jul 22, 2013 (gmt 0)
newpost thread iconhtaccess and banning bots
  code is not working for all bots
25bthom62 4:40 pm on Jul 5, 2013 (gmt 0)
newpost thread iconMerchant Centric Bot
 
3Ken_S 3:40 am on Jun 29, 2013 (gmt 0)
newpost thread iconPossible bot
 
4Readie 9:50 pm on Jun 26, 2013 (gmt 0)
newpost thread iconMSNBot posing as Internet Explorer 7 and not reading robots.txt?
 
11robzilla 2:01 am on Jun 26, 2013 (gmt 0)
newpost thread iconJune 2013 - Blocking Bad Spiders / Bots / User Agents
 
26Imaster 2:15 am on Jun 25, 2013 (gmt 0)
newpost thread iconixquick, duckduckgo, startpage
  IP ranges of these search engines
5motorhaven 10:40 pm on Jun 24, 2013 (gmt 0)
newpost thread iconBad behavior from Microsoft IP
 
11keyplyr 4:06 pm on Jun 24, 2013 (gmt 0)
newpost thread iconBots coming from Peg Tech
  198.2.192.0/18
6lucy24 8:53 pm on Jun 22, 2013 (gmt 0)
newpost thread icongood riddance to bad robots
 
3lucy24 6:25 pm on Jun 17, 2013 (gmt 0)
newpost thread iconRequest Protocol
 
12wilderness 9:28 pm on Jun 13, 2013 (gmt 0)
newpost thread iconFront Range & Pagesinvenotry
 
3Ken_S 9:08 pm on Jun 7, 2013 (gmt 0)
newpost thread iconGoodby old "friends" - Hello Watson
  I.B.M. buys SoftLayer for 2bln
2blend27 11:50 am on Jun 6, 2013 (gmt 0)
newpost thread iconFortinet Technologies
 
3wilderness 11:51 pm on Jun 3, 2013 (gmt 0)
newpost thread iconSwapdrive Norton
 
2wilderness 8:02 pm on Jun 3, 2013 (gmt 0)
newpost thread iconStrange Googlebot behavior
  Grabbing same page over and over in one hit
7grandma_genie 4:57 pm on May 31, 2013 (gmt 0)
newpost thread iconoBot robot where art thou?
 
5lucy24 9:16 pm on May 30, 2013 (gmt 0)
newpost thread iconIANA Country Codes
  No longer published as country codes?
10dstiles 9:22 pm on May 27, 2013 (gmt 0)
newpost thread iconNeedle
  Ouch, here is a thread for you.
11blend27 9:19 pm on May 27, 2013 (gmt 0)
newpost thread iconRackspace
  198.61.128.0 - 198.61.255.255
7Bewenched 9:18 pm on May 27, 2013 (gmt 0)
newpost thread icontrue.net business internet or hosting?
 
7incrediBILL 7:18 pm on May 17, 2013 (gmt 0)
newpost thread iconabove the law? ignores ROBOTS.TXT
 
13lucy24 10:41 am on May 17, 2013 (gmt 0)
Displaying Page 8 of 215<< Back 1 2 3 4 5 6 7 [ 8 ] 9 10 11 12 13 14 15 16 17 ... 215 Next >>
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved