- Search Engines
-- Search Engine Spider and User Agent Identification
- 5:03 am on Jun 24, 2009
It would have been nice if there was a standard plugin/module deployed in all webservers that would auto ban bots that either did not read robots.txt or disobey it.
It's a good idea in theory. In practice it's not doable. Cause with automation it's not always possible to ensure what you're banning is really a bot.
Brought to you by