Forum Moderators: phranque

Message Too Old, No Replies

Bing: Crawl politeness issues

Bing is saying we're "preventing Bing from crawling effectively".

         

rmissey

3:37 pm on Jan 22, 2015 (gmt 0)

10+ Year Member



I keep getting this notification from Bing Webmaster Tools:

The current crawl setting for your site is preventing Bing from crawling effectively. Please make sure you have increased crawl rate setting configured on the Bing Webmaster Tools crawl settings page.


Sounds simple enough, right?

Problem is, I have the throttle open full, all day, and Crawl Boost is turned on. There's no way to give anything more in the crawl settings page.

Any thoughts on what's causing this?

lucy24

8:32 pm on Jan 22, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Is it possible you set a Crawl-Delay line in robots.txt and forgot all about it?

Your robots.txt file will take precedence over the information from this feature.

rmissey

8:43 pm on Jan 22, 2015 (gmt 0)

10+ Year Member



Nope, robots is pretty straight-forward. Few disallow lines, couple allows, sitemap declaration.

Checked the HTTP headers just to make sure somebody didn't sneak it in there & can't find a delay header.

Checked with the technical team & they swear they're not throttling or blocking Bing bots, either.