lammert - 7:20 am on Mar 24, 2005 (gmt 0)

All the talks about the DMCA is of course interesting, but only applies to the US and a few other countries. If I decide my scraper site to be in a country like China chances are very small that I will ever being sued for taking snippets from other sites for my static directory scrapers.

Therefore I think the DMCA discussion is theoretical and ethical correct, but not a real solution for the practical problem of scraper sites on the internet at this moment. I have never heard of a court case against a scraper site, but please correct me if I am wrong.

To get rid of scraper directories effectively, forces that are now present on the internet have to make them unprofitable. In that way scraper builders will automatically find a new way to earn quick money. The forces I can see that have the power to stop scraper sites from making money are mainly search engines, advertisers and ISP's.

Search engines can reduce traffic to these sites by applying strong filters in the SERPs. A good filter will without doubt have collateral damage and some genuine content sites will also be filtered.

Advertisers can stop advertising on this kind of sites, but they will only do this if conversion rates are bad. Until now no advertiser has given any information about conversion rates in this thread, but my guess is that these are not so bad. Afterall, a scraper directory is comparable to the new Adlinks feature of Adsense. With a given set of keywords, a list of advertisments is presented that match that keyword.

ISP's are now using email filter techniques based on central maintained IP lists. If a computer is known to send email spam, its IP will be listed and this can trigger an email router to block a message rather than forward them to a mailbox.

You could think of a comparable system with spam sites. It is not difficult to make a central list of all the domain names that are associated with scraper sites. Then the ISP has to do nothing more than block requests of their users to these domain names. We know this kind of filters exist because they are used for adult content, why not apply them for other site types?