Welcome to WebmasterWorld Guest from 54.225.20.73

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Server Farms - Jan 2017
Continuation of the discussion of hosting company IP ranges[2] ( 1 2 )
53 Ocean10000 3:13 am Oct 18, 2017
At Home with the Robots: 2017 edition
Critique of the Year's Active User Agents
15 lucy24 3:52 pm May 18, 2017
Blocking Methods
How to block access to your server
1 keyplyr 6:29 pm May 10, 2017
Default User Agents of Programming Libraries and Command Line Tools
Resource page for common user agents
15 incrediBILL 8:54 pm Apr 29, 2008
Dataprovider.com
new UA
3 keyplyr 4:04 am Oct 19, 2017
 
0 12:00 am Jan 1, 1970
Gowikibot
8 keyplyr 9:39 am Oct 18, 2017
Mr.4x3 Powered
1 keyplyr 3:15 am Oct 18, 2017
PDFDriveCrawler
1 keyplyr 7:03 pm Oct 14, 2017
Google referers
Not HTTPS
7 dstiles 9:05 pm Oct 13, 2017
Keyword Extractor 5000
1 keyplyr 7:25 pm Oct 12, 2017
CyotekWebCopy
1 keyplyr 7:08 pm Oct 12, 2017
Blank header fields
7 dstiles 6:44 pm Oct 12, 2017
Barkrowler
[2] ( 1 2 )
32 keyplyr 7:59 am Oct 11, 2017
AliCloud ranges
1 keyplyr 9:19 pm Oct 9, 2017
DaveCrawler
1 keyplyr 9:25 pm Oct 8, 2017
XenForo
1 keyplyr 7:29 pm Oct 7, 2017
What is this IP doing
3 born2run 6:50 pm Oct 7, 2017
Postgenomic
1 keyplyr 7:28 pm Oct 6, 2017
DarcyRipper
website ripper/website downloader
5 jonasjacek 8:22 pm Oct 5, 2017
Magic Browser still with us
4 lucy24 4:49 am Oct 4, 2017
Most of Your Traffic is Not Human
[3] ( 1 2 3 )
67 keyplyr 12:48 am Oct 3, 2017
Php/smf
1 keyplyr 7:30 pm Oct 2, 2017
shrinktheweb
new UA, new range
9 keyplyr 8:34 pm Oct 1, 2017
So I am blocking some bots via UA please help.
6 born2run 2:33 am Sep 29, 2017

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members