Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Help me with my .htaccess file please
2 a_angova 3:07 am Sept 8, 2005
  200 (return error code 0) - What does it means?
2 Shaundm 1:56 pm Sept 7, 2005
  envolk[ITS]spider
4 wilderness 1:51 pm Sept 7, 2005
  JRSocketEngine?
4 teasers 3:33 am Sept 7, 2005
  SlurpConfirm404
Is this really Slurp?
4 fish_eye 6:34 pm Sept 6, 2005
  Strange UA
4 wilderness 1:31 am Sept 5, 2005
  Mozilla/4.0 (compatible; prgrabber 1.0)
Is this a Google PageRank checker?
2 GaryK 6:17 pm Sept 4, 2005
  Amazon Crawling
aws-alexa libwww-perl/5.65
9 The_Contractor 5:37 pm Sept 4, 2005
  msnbot/1.0-MM (+http://search.msn.com/msnbot.htm)
Comming from 65.54.164.###
4 caspita 2:39 pm Sept 2, 2005
  Yahoo Spider consuming large amounts of bandwidth
3 servoweb 3:09 pm Sept 1, 2005
  Checkbot and LWP
University uses to check links
2 guitaristinus 3:05 pm Aug 31, 2005
  InfociousBot (+http://corp.infocious.com/tech_crawler.php)
and another bad bot
3 DoppyNL 1:32 pm Aug 30, 2005
  Ocp
4 wilderness 2:08 am Aug 30, 2005
  Yahoo spidering from new Class C subnet
5 volatilegx 1:44 pm Aug 29, 2005
  Unknown spiders visiting
3 Dave_A 10:49 pm Aug 28, 2005
  double slashes
2 wilderness 7:01 pm Aug 28, 2005
  Can you help me with this IP
is it spider
2 tomasz 1:34 am Aug 28, 2005
  Jakarta Commons-HttpClient/3.0-rc1
Anyone seeing this?
18 The_Contractor 5:41 pm Aug 27, 2005
  DynaWeb?
Chinese proxy service
4 keyplyr 5:15 pm Aug 27, 2005
  SquigglebotBot/1.0
Is this related to Relevare?
3 GaryK 5:06 pm Aug 27, 2005
  Jayde
3 wilderness 11:06 am Aug 26, 2005
  YahooSeeker-Testing/v3.9
testing what ... I wonder?
6 bobothecat 1:24 pm Aug 25, 2005
  Oracle Ultra Search
no robots
3 bull 2:43 pm Aug 23, 2005
  Becomebot
4 reaper 3:27 pm Aug 22, 2005
  DTAAgent
3 bumpaw 2:26 am Aug 22, 2005