Welcome to WebmasterWorld Guest from 3.233.226.151

Forum Moderators: mack

Message Too Old, No Replies

Are you taking the MSNbot Gamble?

how much bandwith are you allowing

     
5:22 pm on May 24, 2004 (gmt 0)

Junior Member

joined:Jan 20, 2003
posts:105
votes: 0


The MSNbot has quite a voracious appetite for spidering websites. Some webmasters love it and try to feed it as much as possible. Other webmasters don't see any reason to use up bandwith for a search engine that doesn't currently exist.

Personally, bandwith is cheap for me and I'm willing to feed MSNbot as long as it doesn't impact performance for users.

So how much bandwith are you allowing?

8:23 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 21, 1999
posts:2141
votes: 0


Remember guys, TOS [webmasterworld.com] 4 and 19 applies to ALL of our members, regardless of who employs them. Let's get back on topic!

Fortunately, bandwidth is cheap for my US based sites, so I just let MSNbot get her fill. Maybe that saved some of you a few bytes. ;)

9:34 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member sem4u is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 18, 2002
posts:3086
votes: 5


Welcome to Webmasterworld msndude! :)

It is good to see that another search engine rep has joined the ranks.

11:45 am on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2003
posts:730
votes: 2


Steve, remember that everybody here doesn't have the same level of knowledge of webmastering. Before that episode, I had heard but was completely unfamiliar with robots.txt. So by the time I was aware of the problem, the damage was done. The robot consumed my bandwidth in one weekend.

There is a robots.txt now. So don't assume that I was lazy. Don't make up scenarios when you don't know the full extent of the story.

As for the "rude" part, had I not added that word, you guys wouldn't have perceived my message as rude, just very direct. It's all in the packaging.

However, calling someone "lazy" is rude and disrespectful.

3:32 pm on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 15, 2004
posts:1300
votes: 0


Remember guys, TOS 4 and 19 applies to ALL of our members, regardless of who employs them.

What about items 18 and 20? The search company representatives are not standard posters, they occupy a special position on these forums as far as I can tell, and represent clear commercial interests, but do provide a certain insight into those commercial entities, as long as you bring your salt with you.

4:26 pm on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 22, 2003
posts:1023
votes: 0


Welcome to Webmasterworld msndude!

-George

5:03 pm on June 10, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 28, 2003
posts:88
votes: 0


Welcome to Webmasterworld msndude!
Bring It On! Most of us will be looking forward to your comments & suggestions as the new MSN Search Rolls out!

Thumpcyc

5:04 pm on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 15, 2003
posts:1418
votes: 0


Just a thought on this subject, I wonder if MSN is "testing" their search engine with live web pages instead of cached data.

It would be slow as crap, but I can't think of any other reason why it would do what it's doing. Think about this... they have a cached snapshot of your page which is used for text queries just like Google but maybe they're going a step further and actually comparing the results to the "live" page to make sure that it hasn't changed since they cached it, and that it's not a dead link.

I don't think this model would be very efficient in a production environment - in fact I KNOW it wouldn't - but I can see some advantages for them to run this way while testing.

5:15 pm on June 10, 2004 (gmt 0)

Preferred Member

joined:Apr 22, 2004
posts:528
votes: 0


So its MSN's fault you don't know how to do your job?
5:26 pm on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 19, 2002
posts:1022
votes: 1



If you can't handle this bot, maybe it's time to get on a new server. (Unless of course your site is MICKEY MOUSE!)

Because all of you people blocking it are going to be the same one's starting, "MSN dumped me for no resason!" threads.

5:29 pm on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 16, 2003
posts:1298
votes: 0


Welcome to WebmasterWorld, msndude.
Good to have you here!
This 56 message thread spans 6 pages: 56