keyplyr - 11:04 pm on Dec 29, 2011 (gmt 0)
It [crawl-rate directive] was there for all my sites for many years. They ignored it. In the end, I gave up & removed it. No point in a directive that not a single SE--including MSN, who originated it--followed.
I agree. I also removed it since not one SE followed it. And even though it was correctly written and passed validation, it was cited as the reason why one bot was not following the denied files list.
There's a significant issue with SE's and bot runners respecting the property of webmasters. Sadly, I used to trust people a lot more prior to working on the internet.