Welcome to WebmasterWorld Guest from 34.231.21.123

Forum Moderators: mack

Message Too Old, No Replies

Are you taking the MSNbot Gamble?

how much bandwith are you allowing

     
5:22 pm on May 24, 2004 (gmt 0)

Junior Member

joined:Jan 20, 2003
posts:105
votes: 0


The MSNbot has quite a voracious appetite for spidering websites. Some webmasters love it and try to feed it as much as possible. Other webmasters don't see any reason to use up bandwith for a search engine that doesn't currently exist.

Personally, bandwith is cheap for me and I'm willing to feed MSNbot as long as it doesn't impact performance for users.

So how much bandwith are you allowing?

11:42 pm on June 9, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 30, 2002
posts:2662
votes: 103


Nice to see someone from Microsoft finally taking notice. Any chance of a reply to my e-mail?

Regards...jmcc

12:01 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


Welcome to webmasterworld msndude.

While the folks with lots of low traffic pages, like forums, won't like the aggressive crawling, I sure love it. A complete database is an absolute necessity for MSN to compete with Google, so I hope we will see this sort of aggressive free crawling forever.

12:11 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 15, 2004
posts:1300
votes: 0


msndude, it struck me that the bot may not be properly handling dynamic pages, all my pages are dynamic, and have no last modified date in the header, I suspect most people do not have last modified dates, is it possible that the bot is simply set to grab the pages if either no last modified header or if it hasn't been spidered since last update?

I just put up a three page website 4 days ago and have had 14 msnbot page requests in that time.

since there is no way I'm going to reprogram all my stuff to do proper last modified headers, way too much work for zero payoff on my end, is this something that can be fixed on msn's end?

I definitely welcome the aggressive crawl, although if msn is able to crush google we'll be singing a different song soon enough....

12:18 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 15, 2003
posts:1418
votes: 0


Send the bot my way, I have plenty of bandwidth :)

If MSN's search DOES end up crushing Google I'll be glad I was indexed. And even if it doesn't, there will certainly be enough hype about it in the beginning that it will be worth while.

SEO's are going to have a field day making sites index high in both Google AND MSN though.

1:53 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2000
posts:1133
votes: 0


Msnbot sure knows how to crawl. That's one butt-kickin' spider. Let the games begin!
2:07 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 30, 2002
posts:2662
votes: 103


I'd far prefer to provide Microsoft with some kind of feed that is less bandwidth intensive. One of the reasons for this is that my main site is approximately 80K of static pages. A new site that I am working on has over 1.5 Million static pages of localised UK domain and website information. Having such an aggressive bot as msnbot hitting the bigger site would destroy my bandwidth. The current msnbots are hitting at the rate of approximately 1 page every three seconds. This kind of search partner feed would probably be very attractive for the webmasters of large sites as it would save them on bandwidth charges each month.

It would be a good thing if Microsoft would work with webmasters if it wants to make a superior search engine. However most search engines are inherently algorithmic so that claim does not impress people here. :) (Though I still think that the emphasis on personalisation is wrong. The more important thing for most people I talk to about searching is the localisation aspect and the way Google et al are going about it is wrong - people do not search by postcodes.)

Regards...jmcc

2:18 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2003
posts:730
votes: 2


MSNDUDE, I sent two messages over three weeks ago and I'm still waiting for an answer from your office. I'm a sceptic, so I'll believe you're committed when someone from your team replies to my concerns.

Until then I consider your intervention PR and damage control and that's not what I require or expect from you.

I don't apologize for being rude because I had to pay for the privilege of feeding your experiments last month.

2:28 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 15, 2004
posts:1300
votes: 0


Until then I consider your intervention PR and damage control

that's what I consider all these search engine guys who post here, why would you consider it anything else? I'm going to have to go to the store to buy more salt since you have to take everything the employee of a company says about said company's product with large grains of salt, I think I'll go with rock salt this time.

Especially MS, who don't have a very good track record in the PR area. But I still welcome a new search engine, I'd rather deal with 3 fairly equal ones than 1 main one, which is what I have to do now, google is getting boring with their endless tweaks [ hint: msn, make a reliable, stable product, with reliable, stable results, don't do huge algo tweaks if at all possible... nobody cares about search, they just care about getting reasonably decent results ].

5:03 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 15, 2003
posts:1418
votes: 0


I don't apologize for being rude because I had to pay for the privilege of feeding your experiments last month

Send them the bill :P

5:25 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member steveb is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 20, 2002
posts:4652
votes: 0


"I had to pay for the privilege of feeding your experiments last month."

Translation:

"I can be rude because I'm too lazy to add two lines to robots.txt"

This 56 message thread spans 6 pages: 56