msnbot and IMS

Is msnbot sending If-Modified-Since headers yet?

[Oliver341]

I know this has been mentioned previously but I am hoping for an update. msnbot is currently crawling my site heavily and continously.

I don't mind msnbot indexing my site but if it is going to crawl my site continuously without checking to see if the page has been modified first then I'll have to ban it for bandwidth reasons.

So my question is: does msnbot send If-Modified-Since headers when requesting pages?

[jimbeetle]

Hi Oliver 341,

Not sure of the If-Modified-Since headers, but the robots.txt "crawl-delay" that msnbot recognizes might be of some help to you:

[webmasterworld.com...]

Jim

[Oliver341]

I could slow it down but that's not really what I want. The concept of a hungry robot is a good one, but only when it eats what it needs to eat and not more, if you see what I mean :)

Googlebot knows that it has a hungry bot, and so it uses IMS. Receiving IMS from hungry spider bots is nothing short of mandatory for webmasters with large sites and less than huge bandwidth allowances. Using IMS will mean less webmasters will ban it, and will ultimately make msnbot a more inclusive robot.

[jimbeetle]

Heartily agree with you. Hopefully MSN will get its act together sometime soon and realize that not using If-Modified-Since headers is a drain on resources for all parties.

From what I've seen so far in the new search preview I'm assuming/forecasting (wishing for?) about the same number of referrals from the new search as from MSN/Ink, so I can't afford to ban it, just try to control it a bit.

[sabai]

It definately doesn't use them...

Accept: text/html
Accept-Encoding: identity;q=1.0
From: msnbot(at)microsoft.com
Host: *******.com
User-Agent: msnbot/0.11 (+http://search.msn.com/msnbot.htm)