Welcome to WebmasterWorld Guest from 50.19.156.133

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Puzzled by this UA behavior
5 aristotle 8:13 pm May 26, 2014
Gimme60bot
4 lucy24 8:01 pm May 26, 2014
How to Block Bogus User Agent ") )"
17 wilderness 7:18 pm May 24, 2014
Zeek-heartbleed-notifyer user agent and IP data
3 keyplyr 6:53 pm May 22, 2014
Hit from Homeland Security
13 slipkid 4:17 pm May 20, 2014
Java/1.6.0 04 user agent
5 aristotle 7:23 pm May 19, 2014
u12Bot from Ukraine downloading sites
3 keyplyr 1:22 am May 19, 2014
Squider crawling from many IPs, may be a botnet
major hitter over the past ten days
2 dstiles 9:24 pm May 18, 2014
Riddler honors robots.txt
2 keyplyr 6:19 am May 16, 2014
Elisabot 1.0 user agent and IP information
5 keyplyr 8:28 am May 15, 2014
Inktomi vulnerability probe
WordPress probe
9 keyplyr 1:08 am May 15, 2014
Facebook scraper ? I can't figure this out.
6 kahuna 11:18 pm Apr 30, 2014
Tiscali IstellaBot
3 carson63000 9:17 pm Apr 30, 2014
MSIE 11 odd UA string
like gecko
2 dstiles 2:23 pm Apr 28, 2014
Level 3 - Block 4./8 and 8./8 or not?
How to handle?
15 keyplyr 1:01 pm Apr 28, 2014
bot or preview?
1 lucy24 9:14 pm Apr 23, 2014
Crawling by Google
How google crawling algo works
9 hayden 7:03 am Apr 18, 2014
How To Block Possible Click Bombing
20 EmptyRoom 5:03 am Apr 15, 2014
Can the Googlebot read the blocking rules inside .htaccess?
5 Andy500 8:55 pm Apr 14, 2014
Can Cloudfare stop content scrapers?
4 Andy500 5:12 pm Apr 14, 2014
Ensnare web security tool
Ensnare Attack Detection Tool Hopes to Frustrate Hackers
1 dstiles 8:11 pm Apr 10, 2014
Should I block msnbot-media
5 shamrock 8:26 pm Apr 9, 2014
Previous Server Farms Threads
4 incrediBILL 4:46 pm Apr 9, 2014
Should blank user agents be blocked or not?
5 aristotle 7:47 pm Apr 7, 2014
Server Farms - Feb. 2014
Ongoing Hosting Data Center Discussion[4] ( 1 2 3 4 )
109 incrediBILL 5:43 pm Apr 7, 2014

Featured Threads

Hot Threads This Week

Hot Threads This Month