-- Search Engine Spider and User Agent Identification
---- Server Farms - March 2013
Kendo - 1:49 am on Apr 27, 2013 (gmt 0)
no reason to send requests to our web site servers other than to steal content
Hey, I feel the same pain. You should see our block list. In fact most of our sites run scripts to weed out non-persons-of-interest that traps most bots and scrapers. Well that is, commonly used scrapers that are provided as free open sources of annoyance.
But what about the bots that will create membership on your site, reply to email validation and then log in to scrape member content? Anyone seriously scraping content will have such an app custom designed. I see projects like this advertised on outsourcing services all the time.
If you have forums a heck of a lot of hits will appear to come from genuine surfers who are really using SEO forum spam software that may never be seen coming from the same IP address.