Forum Moderators: mack
Personally, bandwith is cheap for me and I'm willing to feed MSNbot as long as it doesn't impact performance for users.
So how much bandwith are you allowing?
Because all of you people blocking it are going to be the same one's starting, "MSN dumped me for no resason!" threads.
Heh isn't that the truth. Funny how people realize they have the right to block whoever they want from their site, but when someone does it to THEM they get their panties in a twist about it.
Until somebody suggests something more coherent, my first suspicion is that the msn bot is requesting pages repeatedly that do not have a last modified by header, dynamic pages.
This would be simple to determine if everyone who is posting were to check this on their sites now, and might give the msndudes something concrete to work on re fixing the problem. I have something like 40+ sites, all dynamic, all being aggressively spidered, none return last modified headers. Unfortunately, I don't have any non dynamic sites any longer to do a counter check on, I doubt most people here do.
Heh isn't that the truth. Funny how people realize they have the right to block whoever they want from their site, but when someone does it to THEM they get their panties in a twist about it.
During the last week, MSNBOT has downloaded one of my sites 4 (FOUR) times. The site has almost 4000 pages (all unique content) and in my logs there are 16757 page hits by msnbot.
All pages are STATIC, 99% of those pages unmodified since 20-May and my server prints a "Last-Modified:" header, yet msnbot just downloads the whole site with HTTP 200 (rather than 304 as a well behaved Web app would).
Plus, my (perhaps biased, I admit it) opinion is that MS Search will index the Web for free (ie x billion pages listed in our index, where x close to Google's) BUT THE MAJORITY OF THE TRAFFIC WILL GO TO THEIR PPC CLIENTS, because the sponsored listings will come first.
I've not blocked msnbot, but I understand it can be a problem for some, especially if people think there is no benefit.
Same why reports suggest that Y owns 20% of the SE market, yet most of my sites receive 3-9% of their free referrals via Yahoo.
Same why reports suggest that Y owns 20% of the SE market, yet most of my sites receive 3-9% of their free referrals via Yahoo.
Does ANYONE receive more than 3-9% of their traffic from Yahoo? The only time I got a lot of traffic from Yahoo, even being in their paid directory and everything, was with my paid Overture ads. Even then it wasn't a huge percentage.
All pages are STATIC, 99% of those pages unmodified since 20-May and my server prints a "Last-Modified:" header,
thanks, I knew I could confirm or deny that theory very quickly, that denies it, I was giving the msnbot the benefit of the doubt, guess the problem is more fundamental, this would be in keeping with MS's pattern of having version 1 of a new product be essentially useless, primary function just to provide a door into new markets, but I don't see them dropping the search ball, google is too easy of a target right now... of course if ms insists on running the search engine server farm on windows they are going to be a bit handicapped, hotmaiil was never the same since they forced it to run windows and not unix... : )
Exactly. Whether it is laziness or non-comptence, complaining that it is someone else's fault that you aren't doing or don't know how to do your job is way beyond rude.
Ban the bot from specific directories if you want, but this aggressive crawling is just about the best news that webmasters could ask for from MSN at this point. Of course they could completely bungle the ranking of the data, but it sure would be a great thing to have every page that I want crawled to be crawled every day.
