lucy24 - 7:37 am on Jan 16, 2012 (gmt 0)
Has anyone else noticed an absolutely morbid taste for robots.txt in the bingbot? Overall about 3/4 of their pickups are robots.txt. In the last few days I've found stretches where they get 20 or more copies of robots.txt for every one real page.
Looking back, they've always had a taste for robots.txt. Early last year-- I pulled some random chunks of logs to check-- it ran about half and half. But now it's getting to the point where, if the bingbot were a human, I'd ask if they were feeling all right and suggest that they should see a doctor. Of, ahem, one kind or the other.
I had to draw a quick graph to confirm one last hunch. Bing is crawling my site a lot more than they did a year ago. But they're not picking up more pages. That's remained pretty constant. What they're doing is making more and more requests for robots.txt.
Dear bingbot. Is there anything we can do to help you?