moTi - 8:25 pm on Jun 6, 2013 (gmt 0)
fwiw after neglecting my site for over a year, the bingbots started a massive distributed high frequency crawl on my website on monday. never seen that before by other scrapers. since they were shooting down my server several times i had to 403 them.
earnings loss and annoyed visitors were the result. browsing the web, i noticed many other webmaster complaints concerning the bingbot. it's outrageous that microsoft doesn't seem to be able to handle their bots.
and don't come with "but you can slow down crawling by crawl-delay in robots.txt or use bing webmaster tools".. i am not obliged to adjust anything, less than ever at the perpetrator's end just to not fall victim to a ddos attack. this is just plain criminal. others go to jail for that kind of stuff but you get away with it, microsoft?