Are you taking the MSNbot Gamble?

how much bandwith are you allowing

[gniland]

The MSNbot has quite a voracious appetite for spidering websites. Some webmasters love it and try to feed it as much as possible. Other webmasters don't see any reason to use up bandwith for a search engine that doesn't currently exist.

Personally, bandwith is cheap for me and I'm willing to feed MSNbot as long as it doesn't impact performance for users.

So how much bandwith are you allowing?

[jmccormac]

Nice to see someone from Microsoft finally taking notice. Any chance of a reply to my e-mail?

Regards...jmcc

[steveb]

Welcome to webmasterworld msndude.

While the folks with lots of low traffic pages, like forums, won't like the aggressive crawling, I sure love it. A complete database is an absolute necessity for MSN to compete with Google, so I hope we will see this sort of aggressive free crawling forever.

[isitreal]

msndude, it struck me that the bot may not be properly handling dynamic pages, all my pages are dynamic, and have no last modified date in the header, I suspect most people do not have last modified dates, is it possible that the bot is simply set to grab the pages if either no last modified header or if it hasn't been spidered since last update?

I just put up a three page website 4 days ago and have had 14 msnbot page requests in that time.

since there is no way I'm going to reprogram all my stuff to do proper last modified headers, way too much work for zero payoff on my end, is this something that can be fixed on msn's end?

I definitely welcome the aggressive crawl, although if msn is able to crush google we'll be singing a different song soon enough....

[digitalv]

Send the bot my way, I have plenty of bandwidth :)

If MSN's search DOES end up crushing Google I'll be glad I was indexed. And even if it doesn't, there will certainly be enough hype about it in the beginning that it will be worth while.

SEO's are going to have a field day making sites index high in both Google AND MSN though.

[mayor]

Msnbot sure knows how to crawl. That's one butt-kickin' spider. Let the games begin!

[jmccormac]

I'd far prefer to provide Microsoft with some kind of feed that is less bandwidth intensive. One of the reasons for this is that my main site is approximately 80K of static pages. A new site that I am working on has over 1.5 Million static pages of localised UK domain and website information. Having such an aggressive bot as msnbot hitting the bigger site would destroy my bandwidth. The current msnbots are hitting at the rate of approximately 1 page every three seconds. This kind of search partner feed would probably be very attractive for the webmasters of large sites as it would save them on bandwidth charges each month.

It would be a good thing if Microsoft would work with webmasters if it wants to make a superior search engine. However most search engines are inherently algorithmic so that claim does not impress people here. :) (Though I still think that the emphasis on personalisation is wrong. The more important thing for most people I talk to about searching is the localisation aspect and the way Google et al are going about it is wrong - people do not search by postcodes.)

Regards...jmcc

[Harry]

MSNDUDE, I sent two messages over three weeks ago and I'm still waiting for an answer from your office. I'm a sceptic, so I'll believe you're committed when someone from your team replies to my concerns.

Until then I consider your intervention PR and damage control and that's not what I require or expect from you.

I don't apologize for being rude because I had to pay for the privilege of feeding your experiments last month.

[isitreal]

Until then I consider your intervention PR and damage control

that's what I consider all these search engine guys who post here, why would you consider it anything else? I'm going to have to go to the store to buy more salt since you have to take everything the employee of a company says about said company's product with large grains of salt, I think I'll go with rock salt this time.

Especially MS, who don't have a very good track record in the PR area. But I still welcome a new search engine, I'd rather deal with 3 fairly equal ones than 1 main one, which is what I have to do now, google is getting boring with their endless tweaks [ hint: msn, make a reliable, stable product, with reliable, stable results, don't do huge algo tweaks if at all possible... nobody cares about search, they just care about getting reasonably decent results ].

[digitalv]

I don't apologize for being rude because I had to pay for the privilege of feeding your experiments last month

Send them the bill :P

[steveb]

"I had to pay for the privilege of feeding your experiments last month."

Translation:

"I can be rude because I'm too lazy to add two lines to robots.txt"