Welcome to WebmasterWorld Guest from 54.145.209.107

Forum Moderators: mack

Why is the MSNbot so shy?

   
5:21 am on Sep 29, 2010 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Hi all, I wonder if anyone can share some thoughts on how to improve the situation.

I have a site where I see every major SE's bots gobbling up thousands (googlebot - tens of thousands) URLs per day. Both Googlebot and Slurp come from one IP, grab 100-200 URLs in less than a minute, then in couple of minutes come back from another IP, grab another batch and that goes on all day.

Not so with msnbot:

It comes from one IP, grabs 1 (ONE!) URL and leaves. In a few minutes comes back from another IP (all legit, Microsoft-owned) and grabs another 1 (ONE!) URL and so on.

I'm not saying that I like SEs hit my server this much but it's a necessary evil - can't rank for anything if you aren't indexed. But msnbot is just ridiculously slow compared to just any other bot out there.

Is there a good way to entice them to get more URLs in one visit?

I'm trying a few things: I'm pinging Bing whenever the sitemap changes (usually just once a day though), have the site verified with Bing's Webmaster Tools, ping RSS with Pingomatic, even Twit regularly. Bunch of incoming links - apparently enough for G and Y to visit often and get a lot of URLs - what else is there for me to try to get msnbot in the door?
 

Featured Threads

Hot Threads This Week

Hot Threads This Month