Page is a not externally linkable
- WebmasterWorld
-- Webmaster General
---- Thoughts on scraping?


lammert - 2:53 pm on Jun 3, 2010 (gmt 0)


I think they might well even fail to notice they are being scraped.

I don't know about the size of these sites you want to scrape, but one, or a small group of IP addresses downloading a significant number of pages won't go unnoticed, unless these site owners don't analyze the visits to their sites at all.

One simple extra rule in their server config file or firewall and you are out of business, unless you plan to go really blackhat and distribute your requests over a network of anonymous IP addresses.


Thread source:: http://www.webmasterworld.com/webmaster/4146221.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com