-- Search Engine Spider and User Agent Identification
---- At Home with the Robots
wilderness - 12:49 am on Feb 13, 2012 (gmt 0)
Pretty funny from the guy who blocks the globe from a US site, you're the ultimate whitelister if ever there was one!
Bill, Websites for me are a second thought. They merely offer a medium for widgets.
Unfortunately non-US & Can countries (not just bots, but John Q. Publc as well) are the worst offenders for plagiarism and language barriers present simple misinterpretation of very simple TOS. FWIW the worst plagiarizers have to been the Oceanic countries
Why keep bots out?
Ever spent a day or week chasing down copyright violators and wasting time and money hunting down site owners and sending DMCA requests?
I'm not sure when I last explained what I do? For more than a decade, I've been digitizing previously published widget (these particular widgets are a very, very narrow and dying market share) periodicals (most have gone under long ago). The hours behind a computer, scanner and the periodicals are in the tens of thousands of hours. (For the sooth sayers, I do realize original copyright exists, however copyright on the OCR'd digital content also exists (simply because it didn't exist in the past).
Websites serve as a venue to present select widget content. In many instances the realization to the widget people that they have located some widget material that either they've never seen or wasn't previously available is a sever temptation.
Very early on, with website, I'd spend all night putting new pages online, only to retire for then evening and realize (after viewing logs) that 3-4 non_North American bots/people/harvesters had grabbed my entire culmination of work in seconds.
At first communication and compliance was attempted. That had the same effect as urinating in the wind. Denial was was the easiest solution.
Simultaneously, I'm open to immediate review to make exceptions for locales outside the borders of the US & CAN.
About a year ago I was introduced to a widget fan in Hungary and it has been most enjoyable. I have other exceptions in the Scandinavian countries, however the problems with the Euro IP providers making very broad changes in their IP methods to retail customers is simply too overwhelming to keep it with (in three days I've three different Class A's from my friend in Hungary).