Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Stopping scrapers from the get-go


tangor - 1:33 am on Mar 1, 2011 (gmt 0)


In some cases we don't want to reveal all our independent research as to which clueless UA's these folks do. Post it here and they will change it, so keep SOME OF THAT under your hat!

Nutch, Indy Library, libwww-perl and all that other out of the can stuff is okay.

Let's just not make our work harder by saying "Hey, idiot! We found 'this' bwahahaha!" because next week we get 'that'...and it only takes us six months extra to figure 'that' out. Let's just be smart...


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4267704.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com