homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

Why is the MSNbot so shy?

WebmasterWorld Senior Member 5+ Year Member

Msg#: 4208548 posted 5:21 am on Sep 29, 2010 (gmt 0)

Hi all, I wonder if anyone can share some thoughts on how to improve the situation.

I have a site where I see every major SE's bots gobbling up thousands (googlebot - tens of thousands) URLs per day. Both Googlebot and Slurp come from one IP, grab 100-200 URLs in less than a minute, then in couple of minutes come back from another IP, grab another batch and that goes on all day.

Not so with msnbot:

It comes from one IP, grabs 1 (ONE!) URL and leaves. In a few minutes comes back from another IP (all legit, Microsoft-owned) and grabs another 1 (ONE!) URL and so on.

I'm not saying that I like SEs hit my server this much but it's a necessary evil - can't rank for anything if you aren't indexed. But msnbot is just ridiculously slow compared to just any other bot out there.

Is there a good way to entice them to get more URLs in one visit?

I'm trying a few things: I'm pinging Bing whenever the sitemap changes (usually just once a day though), have the site verified with Bing's Webmaster Tools, ping RSS with Pingomatic, even Twit regularly. Bunch of incoming links - apparently enough for G and Y to visit often and get a lot of URLs - what else is there for me to try to get msnbot in the door?


Global Options:
 top home search open messages active posts  

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved