Welcome to WebmasterWorld Guest from 54.226.27.104

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Server Farms - Feb. 2014
Ongoing Hosting Data Center Discussion[4] ( 1 2 3 4 )
109 incrediBILL 5:43 pm Apr 7, 2014
Is this googlebot or not?
Seems this in GBot's Range but doesn't Validate
14 webcentric 2:36 pm Apr 7, 2014
updown tester
Digital Ocean ip ranges
1 Hobbs 10:55 am Apr 7, 2014
How to block googleusercontent.com
blocking google and googleusercontent in htaccess
8 rogerroger 5:33 am Apr 7, 2014
Bots Crawling Pages Blocked by Robots.txt
7 incrediBILL 6:52 am Apr 5, 2014
Is there a list of user-agents to block as of today?
I'm launching a new website soon and want to avoid scrapers
6 shamrock 6:06 am Apr 5, 2014
Hack attempt from a MS range
21 keyplyr 2:32 am Mar 25, 2014
What is Project 5 Research, Inc.
12 HollyWMarshall 2:29 am Mar 25, 2014
Strange crawl
8 wilderness 11:22 am Mar 23, 2014
Semalt
Referrer spamming gone mad.
23 blend27 2:10 am Mar 22, 2014
seegnifybot user agent and IP data
4 keyplyr 11:03 pm Mar 20, 2014
Best Practices for blocking in Windows Server, IIS, .NET environment?
8 webcentric 3:11 am Mar 20, 2014
Threatglass
Barracuda's web site threats database
4 dstiles 12:07 am Mar 20, 2014
Google preview
5 lucy24 10:01 pm Mar 17, 2014
anyone seen the kimono scraper tool yet?
23 londrum 2:00 am Mar 15, 2014
What about your Parked domains.
And bots?[2] ( 1 2 )
44 Angonasec 1:15 pm Mar 13, 2014
Stealth crawler out of Cogent / PSINet 38.99.82.191
3 carson63000 3:00 am Mar 5, 2014
.NET Framework Test Client
HttpWebRequest.UserAgent
4 blend27 7:11 pm Mar 2, 2014
Google-HTTP-Java-Client/1.17.0-rc (gzip)
Anyone seeing this?
4 physics 4:17 pm Feb 23, 2014
Announcing find spiders.pl
Wow.
5 physics 4:14 pm Feb 23, 2014
Googlebot iPhone User Agent
9 lucy24 1:19 pm Feb 17, 2014
applewebkit/534.30 (khtml like gecko) Real or fake?
4 jbrohit 7:38 pm Feb 16, 2014
Fieldtech?
9 lucy24 10:58 pm Feb 13, 2014
Google proxy favicon hits
1 dstiles 7:44 pm Feb 12, 2014
The Whitelist
Key Elements[3] ( 1 2 3 )
86 tangor 11:37 am Feb 12, 2014

Featured Threads

Hot Threads This Week

Hot Threads This Month