new bot: zspider

zspider: new bot, don't obey robots.txt

[Hetta]

I got hit by this user-agent:
"zspider/0.9-dev [feedback.redkolibri.com...]

They picked up about 550 of my pages within 3 hours. A quick read of the url given says: "Our zspider implements the Robots Exclusion Protocol".

Yet they never picked up my robots.txt ... how can they obey that if they don't even go grab it?

They say they're a new search engine. Maybe, maybe not, but I've blocked them now.

[marcs]

I'd say ban for sure. Check the URL in the user-agent. They cache robots.txt info and want you to let them know when it changes?

[Dijkgraaf]

Did you just check for the same user-agent requesting robots.txt or did you also double check by looking for any request for robots.txt coming from the same IP range?
I've noticed some bots UA's when requesting robots.txt don't match the one it used when requesting pages. Probably a seperate thread/machine doing the fetching.