Welcome to WebmasterWorld Guest from 54.144.194.161

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Server Farms - Jan 2017
Continuation of the discussion of hosting company IP ranges
17 Ocean10000 12:12 pm Feb 19, 2017
Blocking Server Ranges
pros & cons
1 keyplyr 3:58 am Oct 16, 2016
Default User Agents of Programming Libraries and Command Line Tools
Resource page for common user agents
15 incrediBILL 8:54 pm Apr 29, 2008
PWorxGSvcs
1 keyplyr 3:35 am Feb 20, 2017
 
0 12:00 am Jan 1, 1970
com.google.GoogleMobile/23.0.0
new UA, probably a scraper but can't identify
3 TorontoBoy 12:51 pm Feb 19, 2017
Amazon IP Ranges
distributed cloud computing
15 not2easy 2:24 am Feb 19, 2017
alexa site audit
2 keyplyr 12:58 am Feb 18, 2017
German Wikipedia Broken Weblinks Bot
tools.wmflabs.org
4 jonasjacek 12:27 pm Feb 17, 2017
Typo3
7 keyplyr 5:43 am Feb 16, 2017
DER-bot
8 jonasjacek 9:10 am Feb 15, 2017
Regression-TB
1 keyplyr 12:26 pm Feb 13, 2017
AntaraPlough
1 keyplyr 3:02 am Feb 12, 2017
AtomicBrowser
1 keyplyr 8:53 pm Feb 11, 2017
TinEye-bot
1 keyplyr 10:46 pm Feb 9, 2017
Microsoft Office Protocol Discovery
5 lucy24 8:02 am Feb 8, 2017
Pixel XL
1 keyplyr 2:35 am Feb 6, 2017
UniversalFeedParser
1 keyplyr 8:07 pm Feb 4, 2017
Tianyu-ktouch
1 keyplyr 12:08 pm Feb 3, 2017
damn404 broken link checker
1 keyplyr 11:46 am Feb 3, 2017
jBrowser
1 keyplyr 10:29 pm Feb 2, 2017
Whoiswebsitebot
3 keyplyr 3:05 am Feb 2, 2017
IBrowse
1 keyplyr 9:58 pm Feb 1, 2017
gutsy
Ubuntu/7.10
3 keyplyr 2:37 am Jan 28, 2017
axios
1 keyplyr 10:58 pm Jan 26, 2017