msnbot

whats the use?

[sidyadav]

Why does msnbot re-index thousands of web pages a day, when the results go nowhere? whats the use? I mean, if it was launching in May or June (2004), it would be a logical thing, but lets face it, rumors say its gonna launch in 2005 or even 2006, then why is it wasting so much of our bandwidth now?

What I have done is currently banned the spider via robots.txt. However, I will remove the ban once the real MSN search launches - in a year or two.

Life's too short to waste MB's of bandwidth for no reason :)

Sid
[note: by re-indexing, I did not mean indexing. Its logical for it index data slowly and to re-index it once in a while (once a month?) but totally wrong for it to re-index it in every day or two.]

[sem4u]

Maybe there is a lot of testing going on behind the scenes? Maybe they are testing out how to make a fresher index than Google? Maybe they will launch sooner than you think?

[sidyadav]

yeah, maybe

Sid

[sem4u]

No-one knows Sid - we just have to wait and see :)

[Chico_Loco]

Why bother banning the bot?

Surely it can't be taking all that much bandwidth?

[sidyadav]

yes, it does not take much bandwidth in a day, but the thing is, it comes everyday to my site, downloads the whole site and leaves. Think long-term. How much bandwidth will it waste, everyday, until it launches in what? 1 year? 2 years?

Waste of bandwidth is much more valued than waste of time. ;)

Sid

[Chris_D]

Sid - I'm with you.

MS = sizzle.

I'm waiting for the steak. Your site is huge Sid - block 'em. See if they actually respect robots.txt

I reckon if its not in msword format - thye'll ignore it anyway!

:)

[mikep]

MSN search may become THE most used search engine in the future. I wouldn't ban msnbot. At some time it will stop reindexing your site everyday.

[sidyadav]

MSN search may become THE most used search engine in the future. I wouldn't ban msnbot. At some time it will stop reindexing your site everyday.

In my original most, I said:
However, I will remove the ban once the real MSN search launches - in a year or two.

I reckon if its not in msword format - thye'll ignore it anyway!

lol, if they don't obey no robots.txt, I'm gonna ban them through .htaccess or IP - thats for sure.

I also see that WebmasterWorld has blocked msnbot, I guess Brett's been witnessing the same situation I am.

Sid