What would the best way to block both of these agents? They just sweep through the site ignoring robots.txt, and i have been adding their ip address to be blocked. I want to deny all user agents in .htaccess, so would the user agent Java catch them all or do I have to be specific for each agent? An example would be?
