bose - 6:11 pm on Feb 20, 2006 (gmt 0)

To those that value their content, it makes no sense to make it available to every bot that crawls out from underneath of every rock.

I went white-list only access a while ago, and have seen no drop in traffic. CPU % usage, however, has dropped from nearly 70% to around 30% and bandwidth usage has dropped by about 38%.

Robots.txt used work, and was the norm during the time when things worked on an honor system. With all these MFA scrapers sites and wannabe SEs out there trying to monetize off someone else's work, putting up an "invitation only" entry sign at the gate makes sense to me.

It is a headache to keep up the whitelist updated with IP lists and UAs to be allowed, and I am no where close to having it as complete/accurate as I would have liked it to be, but it makes no sense anymore for me to let every parasitic nightcrawler out to feast on my original work (my stock in the trade) with impunity.

Sure, I am probably painting with a broad brush, and keeping some good guys from accessing my content unintentionally. That's the collateral damage I am willing to accept.

Interesting discussion.