Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  VisWeb
The Visual Semantic Web?
3 bcolflesh 11:18 pm Jan 11, 2005
  Odp
watch your logs after you subm... er, suggested a site.
7 bull 8:59 pm Jan 10, 2005
  search engine IP lists
2 gomer 9:23 am Jan 9, 2005
  Majestic search engine
Majestic search
6 Dave_A 12:42 pm Jan 7, 2005
  MJ12bot/v0.5.0
MJ12bot is a prototype web-crawling robot.
11 pendanticist 1:11 am Jan 7, 2005
  SlobberDawg
Well, okay, it's ours
3 mcneely 12:43 am Jan 7, 2005
  Is this a known spider?
6 obiku 7:49 am Jan 5, 2005
  Yahoo IP with
Mediapartners-Google/2.1 Agent?
10 jim_w 5:12 am Jan 5, 2005
  Just to let you know of a new web crawling robot.
Kiwi robot
4 Dave_A 5:05 am Jan 5, 2005
  Majestic-12
distributed search engine/spider
3 volatilegx 10:05 pm Jan 3, 2005
  63.251.132.118 - chewed up 7.4GB of bandwidth on my site
4 merrick_lozano 9:31 pm Jan 3, 2005
  "SURF" user agent
A more detailed look...
2 JAB_Creations 1:35 am Jan 3, 2005
  Fast
5 wilderness 5:06 am Jan 2, 2005
  User agent: KMHTTP
Is it Excite Japan, HP or something else?
3 GaryK 5:40 pm Dec 31, 2004
  Blaiz-Bee/1.0
New Cluster Engine?
3 pendanticist 9:16 pm Dec 30, 2004
  Anyone recognize this?
getting crawled like crazy
2 Easy_Coder 4:28 pm Dec 30, 2004
  e-collector Spider
4 joeychgo 10:08 pm Dec 27, 2004
  speedy.net.pe
who is
5 kdawg 8:36 pm Dec 23, 2004
  Faxobot/1.0
coming soon per home page
10 Busynut 6:50 am Dec 22, 2004
  New Exava bot?
BecomeBot@exava.com
18 Reflection 5:57 am Dec 22, 2004
  Microsoft spider?
had site hit from multiple IP's one of which assigned to Microsoft
9 ecomagic 12:51 am Dec 17, 2004
  PlantyNet_WebRobot_V1.9
'Nuther one with an e-mail contact addy...
2 pendanticist 1:01 pm Dec 16, 2004
  htaccess regex syntax
7 mattie 7:12 pm Dec 15, 2004
  StanleyWebSpider/1.0
4 Ankheg 12:34 am Dec 15, 2004
  Netcraft Web Server Survey
What's this one?
9 Rollo 5:28 am Dec 12, 2004