Welcome to WebmasterWorld Guest from 54.226.25.231

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Semalt
Referrer spamming gone mad.
23 blend27 2:10 am Mar 22, 2014
seegnifybot user agent and IP data
4 keyplyr 11:03 pm Mar 20, 2014
Best Practices for blocking in Windows Server, IIS, .NET environment?
8 webcentric 3:11 am Mar 20, 2014
Threatglass
Barracuda's web site threats database
4 dstiles 12:07 am Mar 20, 2014
Google preview
5 lucy24 10:01 pm Mar 17, 2014
anyone seen the kimono scraper tool yet?
23 londrum 2:00 am Mar 15, 2014
What about your Parked domains.
And bots?[2] ( 1 2 )
44 Angonasec 1:15 pm Mar 13, 2014
Stealth crawler out of Cogent / PSINet 38.99.82.191
3 carson63000 3:00 am Mar 5, 2014
.NET Framework Test Client
HttpWebRequest.UserAgent
4 blend27 7:11 pm Mar 2, 2014
Google-HTTP-Java-Client/1.17.0-rc (gzip)
Anyone seeing this?
4 physics 4:17 pm Feb 23, 2014
Announcing find spiders.pl
Wow.
5 physics 4:14 pm Feb 23, 2014
Googlebot iPhone User Agent
9 lucy24 1:19 pm Feb 17, 2014
applewebkit/534.30 (khtml like gecko) Real or fake?
4 jbrohit 7:38 pm Feb 16, 2014
Fieldtech?
9 lucy24 10:58 pm Feb 13, 2014
Google proxy favicon hits
1 dstiles 7:44 pm Feb 12, 2014
The Whitelist
Key Elements[3] ( 1 2 3 )
86 tangor 11:37 am Feb 12, 2014
Server Farms - Sept. 2013
Ongoing Hosting Data Center Discussion[11] ( 1 2 3 4 5 6 7 8 9 10 11 )
327 incrediBILL 12:22 pm Feb 7, 2014
yahoo? Is that you?
21 lucy24 9:46 am Feb 6, 2014
Googlebot-Mobile User Agent To Retire For Smartphones
3 engine 8:15 pm Feb 5, 2014
Target Domain in Useragent string
new to me
4 trintragula 12:31 pm Feb 3, 2014
Do Cell Phone Companies Notify DNSBLs?
I don't think they do.
7 incrediBILL 3:44 am Feb 2, 2014
archive.is
4 wilderness 3:24 am Feb 2, 2014
Google IPs Cannot Be Trusted
Only Googlebot With Round-Trip DNS Validation
12 incrediBILL 10:55 am Jan 31, 2014
WordPress Comment Spam Escalation
Simple SpamBot Blocking Stops Massive Spamming Attack[2] ( 1 2 )
46 incrediBILL 9:39 am Jan 31, 2014
To Block or Not To Block That is The Question
block bots block ua
11 EastTexas 12:44 am Jan 29, 2014

Featured Threads

Hot Threads This Week

Hot Threads This Month