MSNBot and If-Modified-Since

Any sign of support yet?

[encyclo]

I don't deal much with static content, but I have one site which is being quite heavily indexed by msnbot and it doesn't seem to be receiving 304 Not Modified headers for unchanged static pages. I see from this earlier thread from 2004 [webmasterworld.com] that at the time there was no support for If-Modified-Since by the msnbot. Is this still the case now or could it be something in my setup which is serving a 200 instead of a 304?

[jdMorgan]

I don't see any sign of msnbot using If-Modified-Since on any of several static sites checked today.

All 200-OK responses with full filesize byte counts for msnbot, as opposed to 304s for Googlebot and Slurp on pages which haven't been recently updated.

Jim

[encyclo]

Thanks Jim, glad it's not just me - Googebot and Slurp are getting 304s as usual on the site (except for the robots.txt, but I believe that is normal behaviour).

The site gets good referrals from MSN, so I'll probably add a crawl-delay to slow the bot down if it becomes more of a problem. Seems like a waste of bandwidth, but sometimes we have to accept even the most unsophisticated bots to get the visitors. :)

[jd01]

Interesting they would not be checking.

I would guess they have the ability to recognize a 304 and are choosing not to... (If I can figure out how, I am sure they can too.)

Maybe they are working with some type of 'on the way in' data comparrison and a bunch of 304's just slows down the process, so they are not checking? Or, maybe they are comparing actual page to actual page to determine modification date and not relying on us?

My guess would be they will start checking at some point in time, but it could be their system relies on 2 full versions of the page being present and letting us store one copy sure does save a lot of space...

Justin