Welcome to WebmasterWorld Guest from 174.129.96.175

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
u12Bot from Ukraine downloading sites
3 keyplyr 1:22 am May 19, 2014
Squider crawling from many IPs, may be a botnet
major hitter over the past ten days
2 dstiles 9:24 pm May 18, 2014
Riddler honors robots.txt
2 keyplyr 6:19 am May 16, 2014
Elisabot 1.0 user agent and IP information
5 keyplyr 8:28 am May 15, 2014
Inktomi vulnerability probe
WordPress probe
9 keyplyr 1:08 am May 15, 2014
Facebook scraper ? I can't figure this out.
6 kahuna 11:18 pm Apr 30, 2014
Tiscali IstellaBot
3 carson63000 9:17 pm Apr 30, 2014
MSIE 11 odd UA string
like gecko
2 dstiles 2:23 pm Apr 28, 2014
Level 3 - Block 4./8 and 8./8 or not?
How to handle?
15 keyplyr 1:01 pm Apr 28, 2014
bot or preview?
1 lucy24 9:14 pm Apr 23, 2014
Crawling by Google
How google crawling algo works
9 hayden 7:03 am Apr 18, 2014
How To Block Possible Click Bombing
20 EmptyRoom 5:03 am Apr 15, 2014
Can the Googlebot read the blocking rules inside .htaccess?
5 Andy500 8:55 pm Apr 14, 2014
Can Cloudfare stop content scrapers?
4 Andy500 5:12 pm Apr 14, 2014
Ensnare web security tool
Ensnare Attack Detection Tool Hopes to Frustrate Hackers
1 dstiles 8:11 pm Apr 10, 2014
Should I block msnbot-media
5 shamrock 8:26 pm Apr 9, 2014
Previous Server Farms Threads
4 incrediBILL 4:46 pm Apr 9, 2014
Should blank user agents be blocked or not?
5 aristotle 7:47 pm Apr 7, 2014
Server Farms - Feb. 2014
Ongoing Hosting Data Center Discussion[4] ( 1 2 3 4 )
109 incrediBILL 5:43 pm Apr 7, 2014
Is this googlebot or not?
Seems this in GBot's Range but doesn't Validate
14 webcentric 2:36 pm Apr 7, 2014
updown tester
Digital Ocean ip ranges
1 Hobbs 10:55 am Apr 7, 2014
How to block googleusercontent.com
blocking google and googleusercontent in htaccess
8 rogerroger 5:33 am Apr 7, 2014
Bots Crawling Pages Blocked by Robots.txt
7 incrediBILL 6:52 am Apr 5, 2014
Is there a list of user-agents to block as of today?
I'm launching a new website soon and want to avoid scrapers
6 shamrock 6:06 am Apr 5, 2014
Hack attempt from a MS range
21 keyplyr 2:32 am Mar 25, 2014

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month