Are you taking the MSNbot Gamble?

how much bandwith are you allowing

[gniland]

The MSNbot has quite a voracious appetite for spidering websites. Some webmasters love it and try to feed it as much as possible. Other webmasters don't see any reason to use up bandwith for a search engine that doesn't currently exist.

Personally, bandwith is cheap for me and I'm willing to feed MSNbot as long as it doesn't impact performance for users.

So how much bandwith are you allowing?

[DaveAtIFG]

Remember guys, TOS [webmasterworld.com] 4 and 19 applies to ALL of our members, regardless of who employs them. Let's get back on topic!

Fortunately, bandwidth is cheap for my US based sites, so I just let MSNbot get her fill. Maybe that saved some of you a few bytes. ;)

[sem4u]

Welcome to Webmasterworld msndude! :)

It is good to see that another search engine rep has joined the ranks.

[Harry]

Steve, remember that everybody here doesn't have the same level of knowledge of webmastering. Before that episode, I had heard but was completely unfamiliar with robots.txt. So by the time I was aware of the problem, the damage was done. The robot consumed my bandwidth in one weekend.

There is a robots.txt now. So don't assume that I was lazy. Don't make up scenarios when you don't know the full extent of the story.

As for the "rude" part, had I not added that word, you guys wouldn't have perceived my message as rude, just very direct. It's all in the packaging.

However, calling someone "lazy" is rude and disrespectful.

[isitreal]

Remember guys, TOS 4 and 19 applies to ALL of our members, regardless of who employs them.

What about items 18 and 20? The search company representatives are not standard posters, they occupy a special position on these forums as far as I can tell, and represent clear commercial interests, but do provide a certain insight into those commercial entities, as long as you bring your salt with you.

[Alternative Future]

Welcome to Webmasterworld msndude!

-George

[thumpcyc]

Welcome to Webmasterworld msndude!
Bring It On! Most of us will be looking forward to your comments & suggestions as the new MSN Search Rolls out!

Thumpcyc

[digitalv]

Just a thought on this subject, I wonder if MSN is "testing" their search engine with live web pages instead of cached data.

It would be slow as crap, but I can't think of any other reason why it would do what it's doing. Think about this... they have a cached snapshot of your page which is used for text queries just like Google but maybe they're going a step further and actually comparing the results to the "live" page to make sure that it hasn't changed since they cached it, and that it's not a dead link.

I don't think this model would be very efficient in a production environment - in fact I KNOW it wouldn't - but I can see some advantages for them to run this way while testing.

[vabtz]

So its MSN's fault you don't know how to do your job?

[Robino]

If you can't handle this bot, maybe it's time to get on a new server. (Unless of course your site is MICKEY MOUSE!)

Because all of you people blocking it are going to be the same one's starting, "MSN dumped me for no resason!" threads.

[Chndru]

Welcome to WebmasterWorld, msndude.
Good to have you here!