Microsoft Updates Crawler - Supports Compression and If-Modified-Since

[Brett_Tabke]

SearchEngineWorld [searchengineworld.com]

Microsoft Live Search has changed the way its Web crawler application operates when indexing millions of Web sites for inclusion in its search engine, adding features aimed at increasing the efficiency of the process by using compression and other techniques to lower the amount of work required by Web servers, the company announced today

[zekele]

At last they are catching up with some basic functionality! Google and Yahoo have been supporting this for years. It's a rather sad testament to Microsoft's progress that they have taken this long to get to their crawler up to standard.

[phranque]

welcome to WebmasterWorld [webmasterworld.com], zekele!

[balam]

> At last they are catching up

Indeed! I rather like the audacious ending to the article:

Microsoft urges webmasters to make sure their servers are ready to take advantage of the Live Search features added today.

Bill, I don't need urging, I read RFC2616 years ago. Nice to see you're keeping yourself busy, though.

About that article... who is the intended audience? It seems to be for either webmasters or idiots. Given the fact the author failed to explain what a "blog" is - and I'm therefore at a loss - I'd have to assume I'm a webmaster. ;)

---

I'll second the motion - Welcome to WebmasterWorld, zekele!

[blend27]

Now if we could just get them to stop the //search.live.com/results.aspx?q=BIG-KEYWORD&mrt=en-us&FORM=LIVSOP Junk that doubles the amount of traffic from the bot in question - that would really SWELL!

ref: [webmasterworld.com...]