Welcome to WebmasterWorld Guest from 54.82.29.141

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Server Farms - Jan 2018
Continuing discussion of hosting company IP ranges
8 keyplyr 2:55 am Jan 15, 2018
At Home with the Robots: 2017 edition
Critique of the Year's Active User Agents
15 lucy24 3:52 pm May 18, 2017
Blocking Methods
How to block access to your server
1 keyplyr 6:29 pm May 10, 2017
Default User Agents of Programming Libraries and Command Line Tools
Resource page for common user agents
15 incrediBILL 8:54 pm Apr 29, 2008
Sonic the robot
3 lucy24 10:04 pm Jan 18, 2018
 
0 12:00 am Jan 1, 1970
xURL
1 keyplyr 7:15 pm Jan 18, 2018
Spoofed robots.txt requester
Single IP range
2 Ilsevel 7:10 pm Jan 17, 2018
Initial double-quote symbol
5 dstiles 3:21 pm Jan 16, 2018
glindahl-cocrawler
10 keyplyr 7:08 am Jan 16, 2018
Windows 200X
Referrer spam pattern
10 Ilsevel 4:20 am Jan 16, 2018
HyScore
1 keyplyr 3:03 am Jan 16, 2018
uCrawler
4 keyplyr 10:03 am Jan 14, 2018
PulsePoint-Ads.txt-Crawler
1 keyplyr 8:14 pm Jan 13, 2018
SleipnirSiteUpdates
3 lucy24 2:20 pm Jan 12, 2018
Leikibot
1 keyplyr 3:48 am Jan 12, 2018
evc-batch
1 keyplyr 7:50 pm Jan 11, 2018
Desktop
New Pinterest Crawl UA
1 keyplyr 8:59 pm Jan 10, 2018
Discourse Forum Onebox
3 keyplyr 11:30 pm Jan 8, 2018
Zend Http Client
coming from Yahoo
1 keyplyr 8:04 pm Jan 6, 2018
Sprinklr
1 keyplyr 10:06 pm Jan 4, 2018
DnyzBot
3 keyplyr 9:56 pm Jan 4, 2018
Server Farms - Jan 2017
Continuation of the discussion of hosting company IP ranges[3] ( 1 2 3 )
65 Ocean10000 9:44 pm Jan 3, 2018
request.js
1 keyplyr 9:59 pm Jan 2, 2018
ActionExtension
1 keyplyr 9:42 pm Jan 2, 2018

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members