Welcome to WebmasterWorld Guest from 54.198.27.243

Forum Moderated by: Ocean10000 & keyplyr

Search Engine Spider and User Agent Identification


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Server Farms - July 2018
Continuing discussion of hosting company IP ranges[2] ( 1 2 )
53 keyplyr 11:53 pm Aug 19, 2018
At Home with the Robots: 2017 edition
Critique of the Year's Active User Agents
15 lucy24 3:52 pm May 18, 2017
Blocking Methods
How to block access to your server
1 keyplyr 6:29 pm May 10, 2017
User Agents of Programming Libraries and Command Line Tools
Common user agents
14 incrediBILL 8:54 pm Apr 29, 2008
Faraday
New UA, new IP range
6 jonasjacek 12:30 am Aug 20, 2018
 
0 12:00 am Jan 1, 1970
LinkAnalyser
from Wikimedia
1 keyplyr 7:33 pm Aug 19, 2018
e.ventures Investment Crawler
9 keyplyr 7:23 pm Aug 17, 2018
dusterio/link-preview
1 keyplyr 2:06 am Aug 17, 2018
CRMNLCrawlAgent
6 keyplyr 5:37 am Aug 13, 2018
Marketing / Ads related bots
3 Dimitri 11:03 am Aug 12, 2018
zgrab
Researchscan
5 keyplyr 1:06 am Aug 12, 2018
Valve Steam GameOverlay
1 keyplyr 3:22 am Aug 10, 2018
DMVG Site Analyzer
4 lucy24 8:50 pm Aug 9, 2018
aria2
3 TorontoBoy 10:20 am Aug 5, 2018
nsrbot
18 TorontoBoy 12:50 am Aug 5, 2018
PHPCrawl
1 keyplyr 3:06 am Aug 4, 2018
GigablastOpenSource
AKA: G-i-g-a-b-o-t , Gigabot
5 keyplyr 8:45 pm Aug 3, 2018
Funny User Agents
fake browser UAs used by bots
5 lucy24 6:54 pm Aug 1, 2018
Joomla! CMS 3.8 API
9 TorontoBoy 1:29 pm Jul 31, 2018
DemandbasePublisherAnalyzer
1 keyplyr 4:02 am Jul 29, 2018
tracemyfile
8 TorontoBoy 5:14 am Jul 28, 2018
swan
1 keyplyr 8:17 pm Jul 27, 2018
Barkrowler
[2] ( 1 2 )
42 keyplyr 10:07 pm Jul 26, 2018
lua-resty-http
new UA, new range
1 keyplyr 4:29 am Jul 26, 2018