W3CRobot/5.4.0

Does anyone else think this is a fake?

[Dreamquick]

Spotted "W3CRobot/5.4.0 libwww/5.4.0" earlier and am a little curious as I have seen this bot twice, on both occasions it was from Japanese address-space and the only other on-site reference to it talks about Inktomi but with a totally different address space...

I'm also puzzled as I wouldn't expect a real W3C bot to send out broken requests as this one seems to be doing.

IP: 210.255.51.230
05.01.2003 01:34:09

This attempt had a referrer of "//www.w3.org/P3P/compliant_sites" which is obviously a broken URL (but I do have a link from that page).

IP: 219.163.188.218
05.01.2003 14:52:29

This attempt had a referrer of "./", this time a relative rather than absolute path resulting in another broken URL.

- Tony

[littleman]

[w3.org...]

My guess is that it is someone using the freely available bot from w3.org -- they are probably using it misconfigured,. It is a good bot that bad people can use to do naughty things.

[Dreamquick]

Oh well you live and learn - I didn't think that W3C had an their own little robot available for download, I'd rather expected if anything they'd have told you how to build one with a rather over-complicated spec document as they normally do with everything else :)

Thanks littleman!

- Tony

[amznVibe]

that person/bot just hit an entire site of mine too:

visitor from 218-188-163-219.vce-lab.net (219.163.188.218)
6:34:30 PM on Thursday, January 23, 2003.
visitor used W3CRobot/5.4.0 libwww/5.4.0

[219.163.188.218...]

shows this nonsense (can I send this idiot a bandwidth bill? I wish!)

domcrawl : Fast domain crawler
This crawler has been gathering 2,342,057 domain names, and the engine has made 19,209,932 search keys in MySQL table.

domcrawl is a very fast, and robust web robot that automatically gathers domain names. This program is powered by Ruby language and MySQL and libwww.

What for?
Just for fun, it's a great toy! You know gathering something from the web is absolutely interesting. Let's compete each other in gathering domain name! Please notice me if you want to compete with me.