MsnBot slow

How to speed the bot up

[taps]

Msnbot is crawling my site very slowly - only 250 pages a day. Compared to Yahoo (~1500) and Googlebot (2000-10000) it is nearly nothing.

After restructuring parts of my site I'd like to see Msnbot more frequently.

My site is content only, ~50,000 results in Yahoo, 170,000 in Google (should be ~50,000 too, but that's another sad story).

Some pages are very good in serps. Others don't even appear in the index. Maybe Msnbot did not crawl them yet.

So how can I speed up Msnbot?

[stuno259]

I'm having all kinds of trouble trying to get msnbot to behave too. My numbers are about the same with 40,000 visits per day by google, 15,000 from inktomi and a measly 500 from msnbot. My site may somehow be penalized too, when I see how many pages are indexed by MSN, it's only 22, versus 300,000 with inktomi and more than a million with google. Esentially MSN indexes my home page and I rank well for my main topic keywords, but on sub pages, I can't be found.

I have lots of problems with MSN trying to get pages with PHPSESSID=#*$!xx. I redirect those, but I don't see why MSN is even picking up that sort of url in the first place. I don't have any links with phpsess IDs in them and have problems with MSN trying to get old urls that I've been 301 redirecting to the proper place for a long time.

Also, small gripe here. MSNbot doesn't even support gzip transfer encoding. Inktomi and google are smart enough to do this and save on bandwidth, but I guess MSN does care about bandwidth.