Welcome to WebmasterWorld Guest from 54.162.12.134

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Server Farms - February 2015
Tracking and Reporting Data Center IP Ranges[12] ( 1 2 3 4 5 6 7 8 9 10 11 12 )
344 incrediBILL 8:17 am Jul 3, 2015
Default User Agents of Programming Libraries and Command Line Tools
Resource page for common user agents
15 incrediBILL 8:54 pm Apr 29, 2008
Quick primer on identifying bot activity.
And a how to guide to slow and stop scraping
21 Ocean10000 12:48 pm Apr 4, 2008
"Anonymous" Bots
5 aristotle 12:38 am Jul 4, 2015
iFile
iPhone file downloader & file manager
9 keyplyr 11:24 pm Jul 3, 2015
best-seo-solution
Hello semalt, my old friend[2] ( 1 2 )
57 Pfui 7:39 pm Jul 2, 2015
Jorgee
6 Pfui 1:17 pm Jul 2, 2015
GET /xmlrpc.php
Mozilla/5.0 (X11; Linux i686; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
22 keyplyr 7:12 pm Jul 1, 2015
Botnet sources
from spamhaus news blog
4 dstiles 5:59 pm Jun 30, 2015
Secure and other valid proxy sources
An escape from servers[3] ( 1 2 3 )
84 dstiles 11:07 pm Jun 29, 2015
Unblock Today
Public VPN
1 keyplyr 6:53 am Jun 28, 2015
Crawlerbot run by Yelp
1 keyplyr 8:16 am Jun 27, 2015
version 1.7 . of what?
4 lucy24 2:18 am Jun 27, 2015
/trackback/
11 lucy24 4:21 pm Jun 25, 2015
humanoid Chrome
5 lucy24 10:54 pm Jun 24, 2015
SurdotlyBot
9 keyplyr 8:36 am Jun 23, 2015
AppEngine-Google- googleusercontent
1 aristotle 1:34 pm Jun 21, 2015
izsearch.com
Search Engine
1 keyplyr 10:31 am Jun 21, 2015
SkypeUriPreview
8 lucy24 9:44 pm Jun 19, 2015
OpenHoseBot
3 keyplyr 8:15 am Jun 18, 2015
LinqiaScrapeBot
2 keyplyr 5:36 pm Jun 17, 2015
Should I be worried?
Many erroneous requests from Russia/Ukraine
18 Rawrishly 4:56 am Jun 16, 2015
Reverter/2.0 & QuickiWiki/1.0
2 keyplyr 9:50 pm Jun 15, 2015
Haansoft
10 keyplyr 10:01 am Jun 13, 2015
Pocket by Firefox
Browser based social media sharing[2] ( 1 2 )
32 keyplyr 9:06 am Jun 8, 2015

Featured Threads

Hot Threads This Week

Hot Threads This Month