Welcome to WebmasterWorld Guest from 54.167.216.93

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
anyone seen the kimono scraper tool yet?
23 londrum 2:00 am Mar 15, 2014
What about your Parked domains.
And bots?[2] ( 1 2 )
44 Angonasec 1:15 pm Mar 13, 2014
Stealth crawler out of Cogent / PSINet 38.99.82.191
3 carson63000 3:00 am Mar 5, 2014
.NET Framework Test Client
HttpWebRequest.UserAgent
4 blend27 7:11 pm Mar 2, 2014
Google-HTTP-Java-Client/1.17.0-rc (gzip)
Anyone seeing this?
4 physics 4:17 pm Feb 23, 2014
Announcing find spiders.pl
Wow.
5 physics 4:14 pm Feb 23, 2014
Googlebot iPhone User Agent
9 lucy24 1:19 pm Feb 17, 2014
applewebkit/534.30 (khtml like gecko) Real or fake?
4 jbrohit 7:38 pm Feb 16, 2014
Fieldtech?
9 lucy24 10:58 pm Feb 13, 2014
Google proxy favicon hits
1 dstiles 7:44 pm Feb 12, 2014
The Whitelist
Key Elements[3] ( 1 2 3 )
86 tangor 11:37 am Feb 12, 2014
Server Farms - Sept. 2013
Ongoing Hosting Data Center Discussion[11] ( 1 2 3 4 5 6 7 8 9 10 11 )
327 incrediBILL 12:22 pm Feb 7, 2014
yahoo? Is that you?
21 lucy24 9:46 am Feb 6, 2014
Googlebot-Mobile User Agent To Retire For Smartphones
3 engine 8:15 pm Feb 5, 2014
Target Domain in Useragent string
new to me
4 trintragula 12:31 pm Feb 3, 2014
Do Cell Phone Companies Notify DNSBLs?
I don't think they do.
7 incrediBILL 3:44 am Feb 2, 2014
archive.is
4 wilderness 3:24 am Feb 2, 2014
Google IPs Cannot Be Trusted
Only Googlebot With Round-Trip DNS Validation
12 incrediBILL 10:55 am Jan 31, 2014
WordPress Comment Spam Escalation
Simple SpamBot Blocking Stops Massive Spamming Attack[2] ( 1 2 )
46 incrediBILL 9:39 am Jan 31, 2014
To Block or Not To Block That is The Question
block bots block ua
11 EastTexas 12:44 am Jan 29, 2014
Nekstbot
New Polish Search Engine
6 keyplyr 12:27 am Jan 29, 2014
Timestamp in useragent string?
8 trintragula 11:19 pm Jan 28, 2014
Anybody have a clue
10 wilderness 3:57 am Jan 27, 2014
msnbot-media/1.1 generating a ton of 404s
NONE existent JPEGs and GIFs
29 blend27 9:41 pm Jan 18, 2014
Does IP Range Block Via WHM Blacklist Block Same IP Ranges at cPanel
Does IP Block at WHM BruteForce Level Filter Down to Unique Domains on VPS?
12 Webwork 10:40 pm Jan 14, 2014

Featured Threads

Hot Threads This Week

Hot Threads This Month