Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Can you browse the web from Outlook Express?
6 GaryK 4:22 pm Mar 30, 2009
  What is the Amazon/Kindle UA?
5 incrediBILL 7:14 am Mar 29, 2009
  freedir, tags2dir
10 dstiles 10:49 pm Mar 28, 2009
  European Web Archive
europarchive.org
2 dstiles 1:28 am Mar 24, 2009
  HTTP GRANOLA and NetBarrier Firewall
Header tags from Mac
11 dstiles 12:11 am Mar 24, 2009
  Osborne
6 wilderness 10:30 pm Mar 23, 2009
  bot; http://
from 216.158.1.nnn
10 Hobbs 9:21 pm Mar 22, 2009
  block whole countries
best tool to lookup IP ranges
11 smallcompany 3:40 pm Mar 19, 2009
  Bad Bot is google?
Bad bot scraping my site found in Google Cache
2 devil_dog 2:00 am Mar 19, 2009
  Healthbot / Health and Longevity Project
Healthbot/Health_and_Longevity_Project_(HealthHaven.com)
10 dstiles 7:22 am Mar 17, 2009
  Trend Micro WebSurf Prefetcher
3 Samizdata 2:29 am Mar 17, 2009
  Iterasi.com, yet another archiving site
3 incrediBILL 11:15 pm Mar 16, 2009
  How to ID Screen Shot Tools?
Make them ID themselves!
22 incrediBILL 11:11 pm Mar 15, 2009
  Charlotte/0.05
28 jimji 11:04 pm Mar 15, 2009
  Tracing Form Spam
From initial page hit to submitting form
6 dstiles 7:40 pm Mar 15, 2009
  iearthworm - yahoo-related or not?
User agent string "iearthworm@yahoo.com.cn"
3 Marino 4:05 pm Mar 13, 2009
  similarpages dot com
New Nutch?
3 tangor 3:50 am Mar 12, 2009
  Googlebot uses Godaddy.
Update to Earlier Thread
6 keyplyr 11:01 pm Mar 10, 2009
  Surf Knight (bot@surfknight.com)
2 Pfui 11:55 pm Mar 9, 2009
  Msn
15 wilderness 10:39 am Mar 8, 2009
  Firefox 0.8+
4 wilderness 5:25 pm Mar 7, 2009
  BotTracer; (+http://www.informacja.pl) scanning [Your URL here]
Okay so this is just creepy.
5 Pfui 12:26 am Mar 7, 2009
  Safari 4 Beta
Is this the right user agent?
7 GaryK 10:29 pm Mar 6, 2009
  Google OpenSocial agent
Appears to be something new from Google
3 GaryK 8:31 pm Mar 2, 2009
  OpenfosBot (Korea)
4 Pfui 7:21 pm Mar 2, 2009