DeeCee - 7:01 pm on Mar 22, 2012 (gmt 0)

@Staffa,

If it makes you feel better, then it is definitely worth it. In the frustrating fight against bots and scrapers, any minute thing that makes it less frustrating has inherent value. :)

I too treat all "machine visitors" the same way. If they cannot convince me that they add value to me as a site-owner, then they can get a kick in the pants. Whether they are otherwise "valid" or not.

That is one of the frustrations I have with many of the kiddy Twitter Swarmer site/bot owners out there, who think that specifically their "contribution" to the Social Networking mess is the best gift to the world since the Sun.

You go to the "crawler page" listed in their UA. Rather than documenting the value they give to site-owners, in exposure, visitors or other; in short why I should let them in; they instead show basically a copy of their investor presentation. Repeating why their content theft and tracking of the world's content is a good investment. They do not realize that the crawler page is a different audience, and that I am not wanting to invest. Still they want me to allow my paid for network and server bandwidth to be taken away supporting their project and their business.