Forum Moderators: open
Anyone been getting crawled by this chap? (Sorry, can't get hold of IP's from awkwardly big logs)
After reading their site as to why they are spidering, seems they are a baby Google. They can spider three quarters of a billion page in the time it takes google to update its index.
More info on it here [cis.poly.edu]
What's the opinions on robots like this. Get in the index "just in case" or ban it?
It doesn't look like misbehaving to me.
