headsup

[PeteU]

fast is spidering with mozilla ua

[Brett_Tabke]

I trust you Pete, but have you got a log line handy? I'd like to see the ip...

[roscoepico]

IP: 209.67.247.232
UA: Mozilla/4.0 (compatible; MSIE 4.01; Win32)

[littleman]

Name: rdcrawler3.uswal.alltheweb.com
Address: 209.67.247.232
Aliases: 232.247.67.209.in-addr.arpa

[PeteU]

:) :) :)

[Brett_Tabke]

I've heard this a couple of times today, but it doesn't sound like crawler behavior. 4-5 hits at random. Is that what your's was? Or was this a crawl?

[littleman]

I am not sure if they were crawling or just hitting pages that were submitted, but they came in to the same domains many times. Off the top it looked like they were link walking. I checked some individual log files and some domains were hit a couple dozen times.

[PeteU]

Definite crawler behavior except it is not going deep, just homepages of domains and subdomains.
Very similar to what Slurp/si does, at least that is what I observed.

[roscoepico]

ditto:-)

[mark roach]

Yesterday it hit the homepage of over 100 domains that I have been submitting (5 a day) recently. It only visited each domain once. Maybe you saw multiple hits because Fast truncates submitted URLs to just the domain name.

Littleman what did you use to find the name of the spider ?

[Brett_Tabke]

Started looking at logs and found it all over the place:
rdcrawler3.uswal.alltheweb.com 209.67.247.232 MSIE v4.x Windows 95/NT (32-bit)

[littleman]

Mark, I made a little cgi script for convenience. There are 100s of them all over the internet if you do not want to put up your own. There is one here [xav.com]. Or, if you have telnet access to a server you could usually just type in:

nslookup 111.222.333.444
or try
host -v 111.222.333.444

I also have a script you can use [cgi-fun.hypermart.net]. But turn off you javascript or it will drive you batty.

[Brett_Tabke]

Thanks Little, I like that page. bookmarked.