Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Are visitors from mail servers legitimate?
Corporate mail server from medical company.
4 grandma_genie 9:30 pm Jul 7, 2012
GoGuidesSpamTracker
Is this spider legit?
3 grandma_genie 1:26 am Jul 6, 2012
Lipperhey Link Explorer
data mining software
2 keyplyr 8:29 pm Jul 5, 2012
Ezooms the New DotBot?
née DotBot?
16 Pfui 9:21 pm Jul 4, 2012
New Softlayer IP Range
Just heads-up
18 blend27 10:44 pm Jul 2, 2012
what is this POST command and why did it get a 200?
22 dupres01 1:44 pm Jun 30, 2012
Opal Telecom
3 wilderness 8:51 pm Jun 28, 2012
80legs Flooding Site - Ignoring Robots.txt
Literally thousands of requests
9 Andem 3:48 am Jun 28, 2012
Fake user agent strings
How to recognize...[2] ( 1 2 )
50 grandma_genie 5:29 pm Jun 26, 2012
User Agent Spam Attack
2 incrediBILL 5:24 pm Jun 26, 2012
Toolly Robot Used HTML User Agent String!
What a Toolly
3 incrediBILL 5:19 pm Jun 26, 2012
Bot-Seeker
hetzner hosting
2 keyplyr 8:09 pm Jun 23, 2012
SmartLinksAddon
4 wilderness 4:47 pm Jun 21, 2012
Google Site Verification Doesn't Take Rejection Well
5 incrediBILL 9:31 pm Jun 16, 2012
Playing Around With Bad Bots
11 bakedjake 8:00 pm Jun 15, 2012
IPTelligent
spam attempts
4 keyplyr 6:27 pm Jun 12, 2012
Mia research project bot
spider@neofonie.de
3 grandma_genie 8:52 pm Jun 11, 2012
User agent switching
16 btherl 6:05 pm Jun 10, 2012
XL Internet Services
hack attempts
3 keyplyr 11:23 pm Jun 8, 2012
LoneStarBot Doesn't Honor Robots.txt
Yippy Ki Yay
4 incrediBILL 3:56 pm Jun 7, 2012
SingleHop
hack attempts
7 keyplyr 8:58 pm Jun 6, 2012
ISPrime
spam attempts
4 keyplyr 7:41 pm Jun 6, 2012
botnet with same user agent targeting specific pages
29 wilderness 10:52 pm Jun 5, 2012
Forum and Form spammer.
2 g1smd 1:38 am Jun 4, 2012
covert Microsoft crawl
6 keyplyr 12:46 am Jun 3, 2012