Forum Moderators: DixonJones
Looking at my log files I see MSNBot has stepped up its usually greedy pace. Incredibly, we have had over 717,500 hits and 16.6 GB of data transferred to MSNBot this month alone. The next closest bot is Inktomi/Slurp at 19,640 hits and 488.5 MB of data.
I started checking MSNBot's hits a few days ago when their data transfers hit 4 GB...and that was only a few days ago.
Has anyone else experienced this? I'm robots.tx-ing MSNBot out of my system for a while, to cool it down, but I don't know if it will drop me completely or what.
I suspect MSN is indexing and caching multimedia files in preparation for launching their "new" proprietary search service. I can imagine them promising faster delivery of multimedia content as their "killer app" in their attempts to unseat Google and Yahoo/Overture. We have a few media files on our server, and multiple spidering could easily rachet up the bandwidth useage.
MSNBot is costing us money!
I had to chuckle at the title of this topic. I look at it this way, I'll take the free crawl now while it is available. Sooner or later MSN is going to get your money. Might as well take advantage of the free crawling while you can. You can limit which files the bot is crawling so that way you can minimize the bandwidth.
(See above thread) MSNBot is in prototype phase...perhaps not even saving its index for production! I don't mind a spider hitting me, but I guarantee MSNBot didn't find anything new after the first 1GB or so. Timestamps didn't change...no differences in the files it found.
My problem with MSNBot isn't that a spider found my site, but that it's a work-in-progress Microsoft piece of software. "Made" by Microsoft. Y'know..Microsoft. The folks who put out Millenium Edition, XP Home, etc.etc. The folks whose single contribution to the computing world (besides the "font" tag) is a model for making money from crappy products.
There is no reason why any spider should hit any site that hard. It's another example of crappy MS programming, yes, costing me money while they try to figure out what the heck they are doing. And will they offer to repay any of us what they cost us when they start charging money for others to retrieve our content? Not bloody likely.
(I spent a week several years ago being all-expense schmoozed by Microsoft at their lair in Redmond, and I can tell you that I have never seen such a disorganized production stream. They believe, sadly correctly, that market share is more important than quality products. I don't trust this 'bot. Particulary when it is so selfish and disrespectful of the pocketbooks of the people who are feeding it...or rather, paying to feed it.)
I doubt very seriously that they are throwing away any indexed content. When they launch their new search service, they need to be equal to or greater than Google's index size. To do that, will require a lot of spidering between now and then.
If you don't want the bot on your site, then disallow it.
I object to MSNBot costing me money in the same way I object to porn spiders indexing me. (I don't do porn.) They are both objectionable. MSNBot for its arrogance and aggressiveness, and porn spiders for their irrelevance and damage to my rankings in other engines.
You KNOW both are just in it to make money from my work. And neither cares how I feel about it, knowing that there is almost nothing I can do about it besides stay vigilant and block them when I can. But once they've got it...it's too late, and I pay the price for them.
