homepage Welcome to WebmasterWorld Guest from 54.225.24.227
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification

Search Engine Spider and User Agent Identification

Forum Charter, Library, Moderated by: Ocean10000 & incrediBILL Forum Options: Reset Last Read Pointers

PreModerated forum to id search engine spiders

 
new Subject Messages Started by Last Message
newpost thread iconhtaccess and banning bots
  code is not working for all bots
25bthom62 4:40 pm on Jul 5, 2013 (gmt 0)
newpost thread iconMerchant Centric Bot
 
3Ken_S 3:40 am on Jun 29, 2013 (gmt 0)
newpost thread iconPossible bot
 
4Readie 9:50 pm on Jun 26, 2013 (gmt 0)
newpost thread iconMSNBot posing as Internet Explorer 7 and not reading robots.txt?
 
11robzilla 2:01 am on Jun 26, 2013 (gmt 0)
newpost thread iconJune 2013 - Blocking Bad Spiders / Bots / User Agents
 
26Imaster 2:15 am on Jun 25, 2013 (gmt 0)
newpost thread iconixquick, duckduckgo, startpage
  IP ranges of these search engines
5motorhaven 10:40 pm on Jun 24, 2013 (gmt 0)
newpost thread iconBad behavior from Microsoft IP
 
11keyplyr 4:06 pm on Jun 24, 2013 (gmt 0)
newpost thread iconBots coming from Peg Tech
  198.2.192.0/18
6lucy24 8:53 pm on Jun 22, 2013 (gmt 0)
newpost thread icongood riddance to bad robots
 
3lucy24 6:25 pm on Jun 17, 2013 (gmt 0)
newpost thread iconRequest Protocol
 
12wilderness 9:28 pm on Jun 13, 2013 (gmt 0)
newpost thread iconFront Range & Pagesinvenotry
 
3Ken_S 9:08 pm on Jun 7, 2013 (gmt 0)
newpost thread iconGoodby old "friends" - Hello Watson
  I.B.M. buys SoftLayer for 2bln
2blend27 11:50 am on Jun 6, 2013 (gmt 0)
newpost thread iconFortinet Technologies
 
3wilderness 11:51 pm on Jun 3, 2013 (gmt 0)
newpost thread iconSwapdrive Norton
 
2wilderness 8:02 pm on Jun 3, 2013 (gmt 0)
newpost thread iconStrange Googlebot behavior
  Grabbing same page over and over in one hit
7grandma_genie 4:57 pm on May 31, 2013 (gmt 0)
newpost thread iconoBot robot where art thou?
 
5lucy24 9:16 pm on May 30, 2013 (gmt 0)
newpost thread iconIANA Country Codes
  No longer published as country codes?
10dstiles 9:22 pm on May 27, 2013 (gmt 0)
newpost thread iconNeedle
  Ouch, here is a thread for you.
11blend27 9:19 pm on May 27, 2013 (gmt 0)
newpost thread iconRackspace
  198.61.128.0 - 198.61.255.255
7Bewenched 9:18 pm on May 27, 2013 (gmt 0)
newpost thread icontrue.net business internet or hosting?
 
7incrediBILL 7:18 pm on May 17, 2013 (gmt 0)
newpost thread iconabove the law? ignores ROBOTS.TXT
 
13lucy24 10:41 am on May 17, 2013 (gmt 0)
newpost thread iconAmazon AWS Hosts Bad Bots[3] ( 1 2 3 )
  amazonaws.com
88Pfui 11:07 pm on May 16, 2013 (gmt 0)
newpost thread iconAcoonBot (Leaseweb Germany)
  37.58.52.15
7Bewenched 9:44 pm on May 12, 2013 (gmt 0)
newpost thread iconCyberbunker
 
9keyplyr 7:00 am on May 11, 2013 (gmt 0)
newpost thread iconFirefox Browser just passing through
 
3lucy24 6:50 pm on May 8, 2013 (gmt 0)
Displaying Page 8 of 215<< Back 1 2 3 4 5 6 7 [ 8 ] 9 10 11 12 13 14 15 16 17 ... 215 Next >>
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved