Welcome to WebmasterWorld Guest from 54.205.119.93

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Needle
Ouch, here is a thread for you.
11 blend27 9:19 pm May 27, 2013
Rackspace
198.61.128.0 - 198.61.255.255
7 Bewenched 9:18 pm May 27, 2013
true.net business internet or hosting?
7 incrediBILL 7:18 pm May 17, 2013
above the law? ignores ROBOTS.TXT
13 lucy24 10:41 am May 17, 2013
Amazon AWS Hosts Bad Bots
amazonaws.com[3] ( 1 2 3 )
88 Pfui 11:07 pm May 16, 2013
AcoonBot (Leaseweb Germany)
37.58.52.15
7 Bewenched 9:44 pm May 12, 2013
Cyberbunker
9 keyplyr 7:00 am May 11, 2013
Firefox Browser just passing through
3 lucy24 6:50 pm May 8, 2013
Wotbox/2.0 is cheating from DSL hosting
Contains unauthorized content
2 incrediBILL 8:18 pm May 6, 2013
Server Farms - March 2013
Ongoing WMW server farm report[4] ( 1 2 3 4 )
92 wilderness 8:15 pm May 5, 2013
Don't stop fearing the webreaper
6 lucy24 7:53 pm May 4, 2013
Ipod, Ipad Iphone user agents
Any bad conduct coming from such UAs
3 slipkid 3:12 am May 3, 2013
British Library bot
UK law now requires British Library to harvest UK web sites.
3 dstiles 1:56 pm May 1, 2013
How Blocking Impacts Traffic and Other Web Services
2 incrediBILL 8:02 pm Apr 30, 2013
stremorbot
15 lucy24 5:51 pm Apr 22, 2013
hosting company out of Japan
203.189.109.108
3 Bewenched 3:53 am Apr 19, 2013
who are piclab and what do they want with us?
4 lucy24 2:31 am Apr 19, 2013
checks.panopta.com User Agent
Another site monitor
16 grandma_genie 1:52 pm Apr 16, 2013
ip-web-crawler - stupidity never goes out of fashion
8 lucy24 8:54 pm Apr 10, 2013
ScreenerBot
3 Ocean10000 8:28 pm Apr 10, 2013
Fake BingBot
1000s of hits from a Slicehost IP[2] ( 1 2 )
31 Gaia 9:29 pm Apr 3, 2013
facebookexternalhit - Accessing /images/ now?
7 brokaddr 9:36 pm Mar 28, 2013
Are there ANY legitimate visits from AWS?
15 1script 6:30 pm Mar 27, 2013
Detecting human users via user agent
Not looking for 100% accuracy, close enough will do
6 bhonda 2:27 am Mar 25, 2013
msresearch.us
cloaked MSN spawn
3 Pfui 3:25 am Mar 21, 2013

Featured Threads

Hot Threads This Week

Hot Threads This Month