| Why is the MSNbot so shy?
|
1script
msg:4208550 | 5:21 am on Sep 29, 2010 (gmt 0) |
Hi all, I wonder if anyone can share some thoughts on how to improve the situation.
I have a site where I see every major SE's bots gobbling up thousands (googlebot - tens of thousands) URLs per day. Both Googlebot and Slurp come from one IP, grab 100-200 URLs in less than a minute, then in couple of minutes come back from another IP, grab another batch and that goes on all day.
Not so with msnbot:
It comes from one IP, grabs 1 (ONE!) URL and leaves. In a few minutes comes back from another IP (all legit, Microsoft-owned) and grabs another 1 (ONE!) URL and so on.
I'm not saying that I like SEs hit my server this much but it's a necessary evil - can't rank for anything if you aren't indexed. But msnbot is just ridiculously slow compared to just any other bot out there.
Is there a good way to entice them to get more URLs in one visit?
I'm trying a few things: I'm pinging Bing whenever the sitemap changes (usually just once a day though), have the site verified with Bing's Webmaster Tools, ping RSS with Pingomatic, even Twit regularly. Bunch of incoming links - apparently enough for G and Y to visit often and get a lot of URLs - what else is there for me to try to get msnbot in the door?
|
|
