Welcome to WebmasterWorld Guest from 54.158.36.59

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Level 3 - Block 4./8 and 8./8 or not?
How to handle?
15 keyplyr 1:01 pm Apr 28, 2014
bot or preview?
1 lucy24 9:14 pm Apr 23, 2014
Crawling by Google
How google crawling algo works
9 hayden 7:03 am Apr 18, 2014
How To Block Possible Click Bombing
20 EmptyRoom 5:03 am Apr 15, 2014
Can the Googlebot read the blocking rules inside .htaccess?
5 Andy500 8:55 pm Apr 14, 2014
Can Cloudfare stop content scrapers?
4 Andy500 5:12 pm Apr 14, 2014
Ensnare web security tool
Ensnare Attack Detection Tool Hopes to Frustrate Hackers
1 dstiles 8:11 pm Apr 10, 2014
Should I block msnbot-media
5 shamrock 8:26 pm Apr 9, 2014
Previous Server Farms Threads
4 incrediBILL 4:46 pm Apr 9, 2014
Should blank user agents be blocked or not?
5 aristotle 7:47 pm Apr 7, 2014
Server Farms - Feb. 2014
Ongoing Hosting Data Center Discussion[4] ( 1 2 3 4 )
109 incrediBILL 5:43 pm Apr 7, 2014
Is this googlebot or not?
Seems this in GBot's Range but doesn't Validate
14 webcentric 2:36 pm Apr 7, 2014
updown tester
Digital Ocean ip ranges
1 Hobbs 10:55 am Apr 7, 2014
How to block googleusercontent.com
blocking google and googleusercontent in htaccess
8 rogerroger 5:33 am Apr 7, 2014
Bots Crawling Pages Blocked by Robots.txt
7 incrediBILL 6:52 am Apr 5, 2014
Is there a list of user-agents to block as of today?
I'm launching a new website soon and want to avoid scrapers
6 shamrock 6:06 am Apr 5, 2014
Hack attempt from a MS range
21 keyplyr 2:32 am Mar 25, 2014
What is Project 5 Research, Inc.
12 HollyWMarshall 2:29 am Mar 25, 2014
Strange crawl
8 wilderness 11:22 am Mar 23, 2014
Semalt
Referrer spamming gone mad.
23 blend27 2:10 am Mar 22, 2014
seegnifybot user agent and IP data
4 keyplyr 11:03 pm Mar 20, 2014
Best Practices for blocking in Windows Server, IIS, .NET environment?
8 webcentric 3:11 am Mar 20, 2014
Threatglass
Barracuda's web site threats database
4 dstiles 12:07 am Mar 20, 2014
Google preview
5 lucy24 10:01 pm Mar 17, 2014
anyone seen the kimono scraper tool yet?
23 londrum 2:00 am Mar 15, 2014

Featured Threads

Hot Threads This Week

Hot Threads This Month