Here is a new one that hit our site yesterday. They are looking for pictures. Went to the site but it is not open for business yet. They are from Sweden. The only thing it requested so far was robots.txt. I'm going to add them to it as soon as I finish here.
Did they come back, and if they did, were they very agressive? There are some of those image bots that are just lethal. I had one do 100 requests a minute. That isn't bad if you only have a 100 or so graphics, but this was an archive with 8k graphics and icons...whew.
We at picsearch are still developing our software. We are using a load spread technique to ensure that a domain should not be hit more than once every ten minutes.
The service we will provide will be public and free of charge. We are not "stealing" pictures but merely downloading them to create searchable thumbnails. References to the original picture and the page it was indexed from are always provided with the search results.
www.picsearch.org and www.picsearch.org/bot.html will be updated with more information during the day.
For more information, please contact firstname.lastname@example.org