Forum Moderators: mack
Personally, bandwith is cheap for me and I'm willing to feed MSNbot as long as it doesn't impact performance for users.
So how much bandwith are you allowing?
Of course they could completely bungle the ranking of the data, but it sure would be a great thing to have every page that I want crawled to be crawled every day.
I agree, msnbot is the first spider to fully crawl a site I do that has about 1200 pages in a long time. A while ago this thread [webmasterworld.com] caused a huge, and very interesting to follow, uproar in the google forums, but oddly enough google is stuck at
4,285,199,774 web pages
and has been for a while now, that time seems to be spent in getting rid of as many pages as possible from the index, cleaning out pages, not fully spidering sites using various oddly lame sounding excuses.
You'll recall the claim was made that there was this upper limit:
4,294,967,296 (2^32) URLs.
Since I had no real way to know if this claim had any validity, I thought it would be a good idea to just watch google for a while, page count hasn't gone up at all for a while now, but complaints of failing to fully spider sites has skyrocketed over that time. Obviously the fact the count stops several million pages under the upperlimit could be sheer coincidence, but that's a very strange coincidence.
If there is anything to this at all, which I don't have any real way of knowing, MSN's timing could be exceptionally good, this may actually be a weak moment for google.
A badly programmed robot comes during a weekend and consummes 80% of may monthly bandwidth and I'm incompetent because I didn't see it coming?
I'm also incompetent for choosing a host that based on previous years traffic was very suited to the regular activities related to the site?
On top of that, by the time the robot.txt was inplace, I was already well over my monthly allocation. So, again, I'm an idiot for blocking this bot, when I already had to pay extra just so my site wasn't shut down. And in good jest, I'm supposed to let this thing hug my resources indefinitely and continue to pay up, just for a vague hope of making it in some serps to be released God knows when?
Again, how much are you getting paid?
for choosing a host that based on previous years traffic was very suited to the regular activities related to the site
it's not a bad idea to have a host that can support the unexpected, like your site suddenly getting really popular, or getting aggressively spidered, I checked my bandwidth useage, the msnbot definitely jumped it way up, but it never got close to the bandwidth I'm allowed per day. My hosting company doesn't cost that much, but this kind of thing is why I picked them too of course, and is why they are top rated.
I try to get more traffic to my sites, so I'm happy to find one more way happening for more traffic to find me. But since you seem happy with your current status, and the steady traffic you're getting, why not just ban msnbot with robots.txt, it's not that big a deal.
I'm taking this opportunity to transform and improve the very foundation of the site. There's a feason the MSNbot liked the site. It's full of original contents.
Some people commented that those who complain about the robot will probably complain about being listed later.
Again, I find all these scenarios that people make up very insulting. They know zilch and they start all these rumours.
For the record, I have never complained about being unlisted. That's the last of my problems. My site contains original contents and is updated daily. In normal months, 40% of my traffic comes from robots.
If MSN blocks me, there's not much I do about it and there's no point complaining. It's not like they are the only game in town.
Welcome to WebmasterWorld, msndude!
Sid
I would also like to welcome MSNDude even though I don't work for them, I am not using Internet Explorer and nobody is paying me.
(They still own us all though :( )
Dixon.
Everyone else, please leave the specifics at home or take it up with MS directly (just as we are not the Yahoo, Google, or Ask Jeeves conduit, we can't be the MS conduit either).
Anyway, back on topic: I have had MSNBot blocked here for the time being. Until I am confident that MSN has an actual proprietary search engine, I am going to leave it there. We've been hearing about Microsofts new search engine since MS techs first talked about it during the hype up to the release of Windows 98 (that was 1997 folks).
So, I think I will wait as see until we see something besides a spider ;-)
Does ANYONE receive more than 3-9% of their traffic from Yahoo?
I'm getting 30% from Yahoo.
