-- Search Engine Spider and User Agent Identification
---- MSNBot has become a constant Fast-Scraper
AlexK - 10:32 pm on Dec 29, 2011 (gmt 0)
you who are giving the SEs the green light by not having a 'crawl-rate' in your robots.txt
I wish that it were true, Staffa. It was there for all my sites for many years. They ignored it. In the end, I gave up & removed it. No point in a directive that not a single SE--including MSN, who originated it--followed.