Welcome to WebmasterWorld Guest from 54.224.44.168

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Server Farms - Jan 2018
Continuing discussion of hosting company IP ranges[2] ( 1 2 )
54 keyplyr 8:13 pm Apr 25, 2018
At Home with the Robots: 2017 edition
Critique of the Year's Active User Agents
15 lucy24 3:52 pm May 18, 2017
Blocking Methods
How to block access to your server
1 keyplyr 6:29 pm May 10, 2017
Default User Agents of Programming Libraries and Command Line Tools
Resource page for common user agents
15 incrediBILL 8:54 pm Apr 29, 2008
NYU Internet Census
imposter
7 SumGuy 11:59 pm Apr 26, 2018
 
0 12:00 am Jan 1, 1970
DownloaderChrome
9 lucy24 11:34 pm Apr 26, 2018
IP address range of ccbot user agent?
4 born2run 10:53 pm Apr 26, 2018
Leikibot
7 keyplyr 6:52 am Apr 25, 2018
Prudent to block empty user agent visitors?
26 born2run 3:11 am Apr 25, 2018
Buck
7 keyplyr 1:55 am Apr 25, 2018
Most of Your Traffic is Not Human
[3] ( 1 2 3 )
69 keyplyr 1:20 pm Apr 24, 2018
VenusCrawler
4 keyplyr 9:16 pm Apr 23, 2018
SurveyBot
the new improved UA
2 lucy24 10:30 pm Apr 22, 2018
BacklinkCrawler
2 lucy24 9:28 pm Apr 22, 2018
another typo bites the dust
2 lucy24 8:57 pm Apr 22, 2018
The Knowledge AI
5 keyplyr 1:16 am Apr 21, 2018
Yatke
1 keyplyr 8:02 pm Apr 20, 2018
Kudelski Security Innovation
Port Scanner
5 blend27 2:19 am Apr 18, 2018
MauiBot
13 keyplyr 10:49 pm Apr 17, 2018
CORS bot
1 keyplyr 10:59 pm Apr 16, 2018
EU Rejects Robots 'Electronic Persons' Status
9 keyplyr 11:22 pm Apr 14, 2018
IndeedBot
3 TorontoBoy 6:54 am Apr 13, 2018
ltx71
2 keyplyr 4:00 am Apr 12, 2018
Whither Semalt and the like?
reducing list of active crawlers like semalt in htaccess
9 Martin_Potter 7:54 pm Apr 10, 2018