Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: mack
Well, your current record is 102 (one hundred two) consecutive successful requests for robots.txt. This seems a bit over the top :) Is it bing's subtle way of telling me that they don't like the robots.txt and keep hoping it will change? Is bingbot offended because it's lumped under the generic "Hey you!" of "User-Agent: *" instead of getting its very own set of rules?
I hope you meant "discrete". The idea of a discreet URL does make an interesting mental picture, though.
Quick run through the logs confirms that it isn't having any trouble finding pages. Including :: cough, cough :: a fair number of explicit "index.html" that sneaked into various internal links. It's just the infatuation with robots.txt that makes it all look so skewed.