Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  NameProtect.com's NPBot bot hides in plain sight via IP 12.175.0.44
Rarely asks for, never heeds robots.txt
2 Pfui 4:53 pm Apr 26, 2006
  crawl-66-249-66-39.googlebot.com
never seen this one in the host logs
3 hu12 4:04 pm Apr 26, 2006
  Patwebbot
3 bobothecat 12:23 am Apr 26, 2006
  zedzo
2 wilderness 7:49 am Apr 25, 2006
  Bacon Media Bot
Bacon Media Bot
3 ByteEnable 3:57 am Apr 25, 2006
  Was my site crawled?
2 TuanLa1972 2:21 am Apr 25, 2006
  lanshanbot/1.0
2 bobothecat 2:33 am Apr 21, 2006
  msnbot visiting in pairs?
2 bobothecat 11:41 pm Apr 20, 2006
  csci_b659/0.13
3 bull 7:07 pm Apr 20, 2006
  noyona_0_1
3 Staffa 3:34 pm Apr 20, 2006
  WebCopier v4.3? What is it?
Found this in my logs
4 Essex_boy 7:59 pm Apr 19, 2006
  66.151.103.144 & 66.151.103.181
2 hfactor 11:14 am Apr 19, 2006
  Yahoo! Mindset
Didn't ask for robots.txt
8 Pfui 11:07 pm Apr 15, 2006
  LG/U8120/v1.0
ignoring robots, nocache tags
6 keyplyr 2:43 pm Apr 14, 2006
  Strange Yahoo Bot
I think
3 innocbystr 7:31 pm Apr 13, 2006
  What is the name the Yahoo image bot?
or is it a seperate bot?
5 Iczer 2:15 pm Apr 12, 2006
  Interesting from Yahoo!
3 volatilegx 12:29 pm Apr 12, 2006
  64.233.173.67
Google spider
14 Key_Master 3:03 am Apr 11, 2006
  schibstedsokbot
4 bobothecat 9:23 pm Apr 10, 2006
  Google Wireless Transcoder
3 Umbra 6:22 pm Apr 7, 2006
  InfoPath.1
How to block
3 JadedJane 4:28 pm Apr 7, 2006
  Strange spidering
something is trying to find non existing files...
5 netchicken1 12:15 pm Apr 6, 2006
  Comrite/0.7.1
comrot robot identification query
5 fusion5 11:33 pm Apr 3, 2006
  rel="nofollow" confusion
don't follow
7 Drumat5280 8:49 pm Apr 3, 2006
  Spider Design
walking the tightrope
5 BerndH 7:27 pm Apr 3, 2006