Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Zeus and Sqworm
3 coyote 6:53 pm Aug 19, 2003
  Getty Images (206.28.73.1) ignoring robots.txt
Fake user-agent, tries to grab all pages
3 jazzguy 5:01 pm Aug 19, 2003
  Good robots and bad ones - your opinion.
script to help get away bad spiders from our sites.
3 AlexPar 6:22 pm Aug 18, 2003
  UA: Java/1.4.1_03
2 coyote 4:23 pm Aug 18, 2003
  SE's out in force
4 Brett_Tabke 3:51 pm Aug 18, 2003
  Anyone recognise 80.58.6.170
4 decstar 10:08 am Aug 18, 2003
  new directory spider
64.69.79.210
3 Peeress 1:45 pm Aug 17, 2003
  New bot? Dmoz-checker?
4 claus 6:45 am Aug 17, 2003
  QuepasaCreep
anyone have a clue who this is
2 penfold25 7:03 pm Aug 16, 2003
  Need to Block Pesky Spider
markmonitor
9 guillermo5000 10:19 pm Aug 15, 2003
  who am i
yet another bot
6 coyote 9:39 pm Aug 15, 2003
  Someone repeatedly asking for two images that don't exist
They have been hitting me hard requesting them
7 AWildman 8:52 pm Aug 15, 2003
  Deciphering log files
tracking bots
4 Mr_Busby 4:50 pm Aug 15, 2003
  Google?
Not sure.
3 wilderness 3:12 pm Aug 15, 2003
  spider.ilab.sztaki.hu
Seems well behaved
2 WitchLars 3:06 pm Aug 15, 2003
  Multiple IP Access
Who, What, How? Bot?
5 Latigo 3:35 pm Aug 14, 2003
  Pita now is called WebVac
name change
6 webvaccrawler 4:24 pm Aug 13, 2003
  New UA: FavOrg
Favicon/link-checker
2 claus 10:04 pm Aug 12, 2003
  minibot(NaverRobot)/1.0
Who is this?
16 darryl_foshee 6:24 pm Aug 12, 2003
  NaverRobot masquerading as GoogleBot
I don't like it
6 Powdork 3:31 pm Aug 12, 2003
  ia_archiver
archive.org
10 wilderness 3:08 am Aug 12, 2003
  Kasetsart University (Department of Computer Engineering) in Bangkok,
SpiderKU/0.9
3 aaron2b 6:00 pm Aug 11, 2003
  What is readwebpage?
2 BlueSky 4:30 pm Aug 11, 2003
  InternetLinkAgent/3.1
4 rainborick 1:48 pm Aug 11, 2003
  Who or what is WebHiker/1.0
3 viggen 12:38 pm Aug 11, 2003