Welcome to WebmasterWorld Guest from 3.233.226.151

Forum Moderators: mack

Message Too Old, No Replies

Are you taking the MSNbot Gamble?

how much bandwith are you allowing

     
5:22 pm on May 24, 2004 (gmt 0)

Junior Member

joined:Jan 20, 2003
posts:105
votes: 0


The MSNbot has quite a voracious appetite for spidering websites. Some webmasters love it and try to feed it as much as possible. Other webmasters don't see any reason to use up bandwith for a search engine that doesn't currently exist.

Personally, bandwith is cheap for me and I'm willing to feed MSNbot as long as it doesn't impact performance for users.

So how much bandwith are you allowing?

9:00 pm on June 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 15, 2004
posts:1300
votes: 0


Of course they could completely bungle the ranking of the data, but it sure would be a great thing to have every page that I want crawled to be crawled every day.

I agree, msnbot is the first spider to fully crawl a site I do that has about 1200 pages in a long time. A while ago this thread [webmasterworld.com] caused a huge, and very interesting to follow, uproar in the google forums, but oddly enough google is stuck at
4,285,199,774 web pages

and has been for a while now, that time seems to be spent in getting rid of as many pages as possible from the index, cleaning out pages, not fully spidering sites using various oddly lame sounding excuses.

You'll recall the claim was made that there was this upper limit:
4,294,967,296 (2^32) URLs.

Since I had no real way to know if this claim had any validity, I thought it would be a good idea to just watch google for a while, page count hasn't gone up at all for a while now, but complaints of failing to fully spider sites has skyrocketed over that time. Obviously the fact the count stops several million pages under the upperlimit could be sheer coincidence, but that's a very strange coincidence.

If there is anything to this at all, which I don't have any real way of knowing, MSN's timing could be exceptionally good, this may actually be a weak moment for google.

1:21 am on June 11, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2003
posts:730
votes: 2


To all those who said I didn't do my job or that I'm incompetent, how much did Microsoft paid you?

A badly programmed robot comes during a weekend and consummes 80% of may monthly bandwidth and I'm incompetent because I didn't see it coming?

I'm also incompetent for choosing a host that based on previous years traffic was very suited to the regular activities related to the site?

On top of that, by the time the robot.txt was inplace, I was already well over my monthly allocation. So, again, I'm an idiot for blocking this bot, when I already had to pay extra just so my site wasn't shut down. And in good jest, I'm supposed to let this thing hug my resources indefinitely and continue to pay up, just for a vague hope of making it in some serps to be released God knows when?

Again, how much are you getting paid?

1:56 am on June 11, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 15, 2004
posts:1300
votes: 0


for choosing a host that based on previous years traffic was very suited to the regular activities related to the site

it's not a bad idea to have a host that can support the unexpected, like your site suddenly getting really popular, or getting aggressively spidered, I checked my bandwidth useage, the msnbot definitely jumped it way up, but it never got close to the bandwidth I'm allowed per day. My hosting company doesn't cost that much, but this kind of thing is why I picked them too of course, and is why they are top rated.

I try to get more traffic to my sites, so I'm happy to find one more way happening for more traffic to find me. But since you seem happy with your current status, and the steady traffic you're getting, why not just ban msnbot with robots.txt, it's not that big a deal.

2:14 am on June 11, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2003
posts:730
votes: 2


Hi. It's already banned. I'm also switching host, but I'm can't do it on a whim.

I'm taking this opportunity to transform and improve the very foundation of the site. There's a feason the MSNbot liked the site. It's full of original contents.

Some people commented that those who complain about the robot will probably complain about being listed later.

Again, I find all these scenarios that people make up very insulting. They know zilch and they start all these rumours.

For the record, I have never complained about being unlisted. That's the last of my problems. My site contains original contents and is updated daily. In normal months, 40% of my traffic comes from robots.

If MSN blocks me, there's not much I do about it and there's no point complaining. It's not like they are the only game in town.

4:37 am on June 11, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 11, 2003
posts:955
votes: 0


Before I banned it, MSNbot consumed 1.4GB in a week. I've blocked it as for now, will let it in once again (if) when it launches - can't care to risk it :)

Welcome to WebmasterWorld, msndude!

Sid

1:14 pm on June 11, 2004 (gmt 0)

Senior Member

joined:Mar 8, 2002
posts:2897
votes: 0


I have to ask you all to re-read the Terms of service section 4 [webmasterworld.com] again please and keep the conversation civilised and on topic.

I would also like to welcome MSNDude even though I don't work for them, I am not using Internet Explorer and nobody is paying me.

(They still own us all though :( )

Dixon.

2:36 pm on June 11, 2004 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38257
votes: 115


Welcome to the board MSN Dude'ster. I think we all know what you can and can't cover here and it is nice to have msn onboard. (I hope your skin is as thick as your spider is aggressive. 'course, you work for MS, you'd have to have thick skin by now. lol!)

Everyone else, please leave the specifics at home or take it up with MS directly (just as we are not the Yahoo, Google, or Ask Jeeves conduit, we can't be the MS conduit either).

Anyway, back on topic: I have had MSNBot blocked here for the time being. Until I am confident that MSN has an actual proprietary search engine, I am going to leave it there. We've been hearing about Microsofts new search engine since MS techs first talked about it during the hype up to the release of Windows 98 (that was 1997 folks).

So, I think I will wait as see until we see something besides a spider ;-)

3:19 pm on June 11, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:May 10, 2004
posts:61
votes: 0


Like everyone else I've been seeing a lot of the msnbot recently. As I have several large sites, and comparatively little traffic, it's been my best "customer" for some months now. (A close second is Ask Jeeves/Teoma, followed by googlebot and in a poor fourth position Slurp). As I have lots of free bandwidth it's not a problem - yet. However: If and when the site grows in popularity, and bandwidth becomes an issue, and there's still no sign of a proprietary MSN search, there's a spot in my robots.txt reserved for msnbot ;-).
4:03 pm on June 11, 2004 (gmt 0)

Administrator

WebmasterWorld Administrator rogerd is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 2, 2000
posts:9687
votes: 1


Hey, welcome msndude... Looking forward to some interesting discussion in the coming months. (I haven't slammed the door on msnbot yet, btw, though I'll be watching its eating habits.)
5:05 pm on June 11, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 2, 2003
posts:1184
votes: 0


digitalv

Does ANYONE receive more than 3-9% of their traffic from Yahoo?

I'm getting 30% from Yahoo.

This 56 message thread spans 6 pages: 56