Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  customer-reverse-entry
huh?
5 Bewenched 4:25 am Sept 26, 2007
  Bots trying to find phpMyAdmin
5 SEOPTI 11:12 pm Sept 22, 2007
  Mediapartners-Google from non-Google IP
Transparent spoof of a Googlebot
2 jdMorgan 9:27 pm Sept 22, 2007
  Sogou
chinese bot
3 Bewenched 12:56 am Sept 22, 2007
  Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
5 SEOPTI 2:41 pm Sept 19, 2007
  Yeti
4 keyplyr 1:37 pm Sept 19, 2007
  Help identifying -who are these guys
identifies itself as Danger
4 Bewenched 2:30 am Sept 13, 2007
  rv:1.8.1 - voila bot?
2 jake66 1:06 am Sept 11, 2007
  Who have Spider Stats? I Need Thanks~~
2 hostsoftware 8:28 pm Sept 4, 2007
  Mediapartners-Google
Has Google changed the name to eliminate the version #?
12 GaryK 6:33 am Sept 1, 2007
  Why don't bots send a special header?!
8 mikomido 11:39 pm Aug 30, 2007
  EchO! Spider Agent
2 jshpik1 2:46 pm Aug 28, 2007
  Ask with no user agent
lots of hits
5 Bewenched 1:36 pm Aug 24, 2007
  PageFetcher-Google-CoOp;
6 keyplyr 8:57 pm Aug 22, 2007
  ASK with no reverse DNS and no UA
13 Romeo 3:29 am Aug 21, 2007
  Spider running from Qwest
7 btherl 1:27 am Aug 21, 2007
  compute.amazonaws.com
sucked down entire site - maybe twice
2 amznVibe 9:31 pm Aug 17, 2007
  ozzie
3 wilderness 1:47 pm Aug 17, 2007
  France Telecom?
but really resolves to Performance Systems International Inc.
11 Bewenched 7:23 am Aug 6, 2007
  Gigabot
not identifying itself?
2 Bewenched 5:41 pm Aug 4, 2007
  Is this googlebot or mediapartners?
Same IP, different host resolution?
3 johnnie 5:41 pm Aug 4, 2007
  http://127.0.0.1:4664/preview?event id=
5 frontpage 12:29 am Aug 3, 2007
  HTMLParser/1.6
from research27.corp.sp1.yahoo.com
4 blend27 8:43 pm Aug 2, 2007
  What am I doing wrong guys to block IP,s
rDNSbot help re - IP,s
2 whatsdoin 2:05 pm Aug 2, 2007
  IBM Supercomputer : Blueice
Scraping my site
5 RayWatson 6:24 pm July 30, 2007