Welcome to WebmasterWorld Guest from 54.159.179.132

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Server Farms - February 2015
Tracking and Reporting Data Center IP Ranges[9] ( 1 2 3 4 5 6 7 8 9 )
256 incrediBILL 5:50 pm May 22, 2015
Default User Agents of Programming Libraries and Command Line Tools
Resource page for common user agents
15 incrediBILL 8:54 pm Apr 29, 2008
Quick primer on identifying bot activity.
And a how to guide to slow and stop scraping
21 Ocean10000 12:48 pm Apr 4, 2008
New Click Adsense
Russian scam bot
20 keyplyr 11:09 pm May 20, 2015
AntiCrawler
New bot crawler
16 dstiles 10:29 pm May 20, 2015
Domain Re-Animator Bot
Zombie apocalypse...
4 trintragula 2:33 am May 20, 2015
New ip class for googlebot ?
193.235.238.233
3 adresanet 1:16 am May 20, 2015
webkit2png
Advanced/Full ScreenShot Maker -command line tool
1 blend27 7:20 pm May 19, 2015
Secure and other valid proxy sources
An escape from servers[3] ( 1 2 3 )
77 dstiles 1:45 pm May 19, 2015
Amazon AWS Hosts Bad Bots
Continuation Thread[4] ( 1 2 3 4 )
108 incrediBILL 7:25 am May 18, 2015
SputnikBot
could be worse...
3 trintragula 7:09 pm May 17, 2015
Scraping May Be A Biological Imperative
Scraping is actually a TREME according to Stephen Hawking
11 incrediBILL 7:26 am May 16, 2015
Findxbot
new bot on the hunt
17 dupres01 8:59 pm May 13, 2015
OmniBot
Microsoft Corp.
12 Pfui 8:06 pm May 13, 2015
Googlebot-Video/1.0"
audio & video files
5 keyplyr 11:25 pm May 11, 2015
PageGlimpse
Another blatant scraper
12 blend27 5:27 pm May 11, 2015
A new UA from Yahoo
YahooMobileHomerun
7 slipkid 2:54 am May 11, 2015
FieldTech redux
1 lucy24 8:38 pm May 10, 2015
Bingbot odd request
11 mrtonyg 2:24 am May 10, 2015
Applebot
Fresh from Apple
18 Pfui 9:12 pm May 6, 2015
Fortinet
8 slipkid 7:48 pm May 4, 2015
github.com/typhoeus
29 aristotle 2:24 pm May 3, 2015
PlurkBot
Taipei; plurk.com
1 Pfui 12:40 am May 1, 2015
BingLocalSearchBot
yet another from MSN
1 Pfui 6:52 pm Apr 30, 2015
bingbot
.ntwk.msn.net
1 Pfui 6:46 pm Apr 30, 2015

Featured Threads

Hot Threads This Week

Hot Threads This Month