pocketseo - 6:01 pm on Jun 21, 2008 (gmt 0)

Maybe there should be a law against not respecting robots.txt directives.

This is a waste of bandwidth and it's corrupting business data/stats.

I blocked msnbot with robots.txt on one personal site as an experiment and it feels great. Traffic is higher than ever because Live search is relatively insignificant.