Welcome to WebmasterWorld Guest from 54.145.235.72

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Server Farms - February 2015
Tracking and Reporting Data Center IP Ranges[6] ( 1 2 3 4 5 6 )
154 incrediBILL 9:32 am Apr 25, 2015
Default User Agents of Programming Libraries and Command Line Tools
Resource page for common user agents
15 incrediBILL 8:54 pm Apr 29, 2008
Quick primer on identifying bot activity.
And a how to guide to slow and stop scraping
21 Ocean10000 12:48 pm Apr 4, 2008
Secure and other valid proxy sources
An escape from servers[2] ( 1 2 )
46 dstiles 10:26 am Apr 26, 2015
github.com/typhoeus
9 aristotle 5:13 pm Apr 25, 2015
Success in getting annoying someone/bot stopped
Hits coming from Cablevision IP
13 slipkid 5:24 pm Apr 20, 2015
Facebook app?
4 lucy24 4:05 pm Apr 18, 2015
Amazon AWS Hosts Bad Bots
Continuation Thread[4] ( 1 2 3 4 )
97 incrediBILL 10:19 am Apr 18, 2015
CrazyWebCrawler
3 lucy24 1:32 am Apr 18, 2015
cyren.com
8 keyplyr 3:15 am Apr 17, 2015
best-seo-solution
Hello semalt, my old friend
30 Pfui 9:29 pm Apr 14, 2015
=Mozilla/5.0
17 Pfui 8:55 am Apr 12, 2015
InfoUSA/InfoGroup
2 keyplyr 9:19 pm Apr 10, 2015
megaindex.ru
1 lucy24 8:43 pm Apr 9, 2015
Qwantify
qwant.com
17 Pfui 7:45 pm Apr 7, 2015
yoozBot
Iran
2 Pfui 11:00 am Apr 7, 2015
IZSearch
10 keyplyr 3:51 pm Apr 5, 2015
fossmovement
2 keyplyr 2:03 am Apr 4, 2015
Ceptro
Brazil
1 keyplyr 10:16 am Mar 30, 2015
Applebot
Fresh from Apple
16 Pfui 11:53 pm Mar 28, 2015
bingbot/3.0-alpha
11 Angonasec 8:18 am Mar 28, 2015
TosCrawler
toshiba
15 keyplyr 1:58 am Mar 28, 2015
alooflybot
aloofly.com
4 Pfui 9:12 pm Mar 27, 2015
pt-PT
12 trintragula 8:33 pm Mar 24, 2015
Netscan
1 Angonasec 12:43 pm Mar 23, 2015

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month