Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Linux and Android
6 wilderness 8:20 pm July 13, 2012
  Education Networks of America
12 incrediBILL 2:39 am July 13, 2012
  Global Net Access
5 wilderness 12:43 am July 10, 2012
  okhy11
007, okhy eleven...
11 lucy24 3:16 am July 9, 2012
  Are visitors from mail servers legitimate?
Corporate mail server from medical company.
4 grandma_genie 9:30 pm July 7, 2012
  GoGuidesSpamTracker
Is this spider legit?
3 grandma_genie 1:26 am July 6, 2012
  Lipperhey Link Explorer
data mining software
2 keyplyr 8:29 pm July 5, 2012
  Ezooms the New DotBot?
née DotBot?
16 Pfui 9:21 pm July 4, 2012
  New Softlayer IP Range
Just heads-up
18 blend27 10:44 pm July 2, 2012
  what is this POST command and why did it get a 200?
22 dupres01 1:44 pm June 30, 2012
  Opal Telecom
3 wilderness 8:51 pm June 28, 2012
  80legs Flooding Site - Ignoring Robots.txt
Literally thousands of requests
9 Andem 3:48 am June 28, 2012
  Fake user agent strings
How to recognize...[2] ( 1 2 )
50 grandma_genie 5:29 pm June 26, 2012
  User Agent Spam Attack
2 incrediBILL 5:24 pm June 26, 2012
  Toolly Robot Used HTML User Agent String!
What a Toolly
3 incrediBILL 5:19 pm June 26, 2012
  Bot-Seeker
hetzner hosting
2 keyplyr 8:09 pm June 23, 2012
  SmartLinksAddon
4 wilderness 4:47 pm June 21, 2012
  Google Site Verification Doesn't Take Rejection Well
5 incrediBILL 9:31 pm June 16, 2012
  Playing Around With Bad Bots
11 bakedjake 8:00 pm June 15, 2012
  IPTelligent
spam attempts
4 keyplyr 6:27 pm June 12, 2012
  Mia research project bot
spider@neofonie.de
3 grandma_genie 8:52 pm June 11, 2012
  User agent switching
16 btherl 6:05 pm June 10, 2012
  XL Internet Services
hack attempts
3 keyplyr 11:23 pm June 8, 2012
  LoneStarBot Doesn't Honor Robots.txt
Yippy Ki Yay
4 incrediBILL 3:56 pm June 7, 2012
  SingleHop
hack attempts
7 keyplyr 8:58 pm June 6, 2012