kuloko-bot crawling hard

[creative craig]

66.90.81.41 - - [20/Oct/2003:23:19:51 +0100] "GET /xxxx.html HTTP/1.1" 206 6624 "-" "kuloko-bot/0.2"

Had this spider come by and take a large portion of my site yesterday evening. It came by the day before and read robots.txt file before it went to work.

kuloko.com still reads about contextual search coming soon!

Anyone have any other news on it?

Craig

[Brett_Tabke]

It looks like a one man shop...

[wilderness]

I have both the UA and the entire server range denied.
Of course, most everybody knows I never lean towards the lenient sider :)

How is it possible to make a determination of what the bot and the site intend to do with the data of which they are spidering, perhaps even mining with no pages explaining the sites goals?
It seems to me, an unreasonable expectation by the bot's owner!
Of course most bot creators/owners/users fail to comprehend the fact that they are invading, possibly without permission or desire at the expense of the webmaster they are visiting.
Perhaps someday there will exist an honored protocol by bots?

Don

[creative craig]

This site at the moment is pretty open, but when have the time I will implement a better robots.txt file.

[bull]

I will implement a better robots.txt file.

Be sure to give kuloko enough time to fetch your new robots.txt (entire log pattern):

66.227.104.196 - - [28/Sep/2003:10:53:44 +0200] "GET / HTTP/1.1" 200 2204 www.-.net "-" "kuloko-bot/0.2" "-"
66.227.104.196 - - [[b]28/Sep/2003:23:59:10[/b] +0200] "GET /robots.txt HTTP/1.1" 200 832 www.-.net "-" "kuloko-bot/0.2" "-"
66.90.81.41 - - [[b]14/Oct/2003:06:51:48[/b] +0200] "GET /robots.txt HTTP/1.1" 200 880 www.-.net "-" "kuloko-bot/0.2" "-"
66.90.81.41 - - [[b]22/Oct/2003:11:13:41[/b] +0200] "GET /start.html HTTP/1.1" 206 621 www.-.net "-" "kuloko-bot/0.2" "-"

[Josk]

Also:

218.20.60.211
195.170.15.134