Welcome to WebmasterWorld Guest from 54.80.8.44

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Server Farms - Jan 2018
Continuing discussion of hosting company IP ranges[2] ( 1 2 )
53 keyplyr 10:31 pm Apr 22, 2018
At Home with the Robots: 2017 edition
Critique of the Year's Active User Agents
15 lucy24 3:52 pm May 18, 2017
Blocking Methods
How to block access to your server
1 keyplyr 6:29 pm May 10, 2017
Default User Agents of Programming Libraries and Command Line Tools
Resource page for common user agents
15 incrediBILL 8:54 pm Apr 29, 2008
Leikibot
5 keyplyr 7:53 pm Apr 24, 2018
 
0 12:00 am Jan 1, 1970
Prudent to block empty user agent visitors?
25 born2run 6:26 pm Apr 24, 2018
Most of Your Traffic is Not Human
[3] ( 1 2 3 )
69 keyplyr 1:20 pm Apr 24, 2018
VenusCrawler
4 keyplyr 9:16 pm Apr 23, 2018
SurveyBot
the new improved UA
2 lucy24 10:30 pm Apr 22, 2018
BacklinkCrawler
2 lucy24 9:28 pm Apr 22, 2018
another typo bites the dust
2 lucy24 8:57 pm Apr 22, 2018
The Knowledge AI
5 keyplyr 1:16 am Apr 21, 2018
Yatke
1 keyplyr 8:02 pm Apr 20, 2018
Buck
6 keyplyr 11:03 pm Apr 19, 2018
Kudelski Security Innovation
Port Scanner
5 blend27 2:19 am Apr 18, 2018
MauiBot
13 keyplyr 10:49 pm Apr 17, 2018
CORS bot
1 keyplyr 10:59 pm Apr 16, 2018
EU Rejects Robots 'Electronic Persons' Status
9 keyplyr 11:22 pm Apr 14, 2018
IndeedBot
3 TorontoBoy 6:54 am Apr 13, 2018
ltx71
2 keyplyr 4:00 am Apr 12, 2018
Whither Semalt and the like?
reducing list of active crawlers like semalt in htaccess
9 Martin_Potter 7:54 pm Apr 10, 2018
Cliqzbot
12 Bewenched 1:16 am Apr 7, 2018
OutclicksBot
6 keyplyr 1:39 am Apr 4, 2018
SitemapParser-VIPnytt
3 keyplyr 8:37 pm Apr 2, 2018