IITian - 11:15 pm on Mar 30, 2004 (gmt 0)

I have a site that had about 50 'starter' pages that were crawled regularly by inktomi. In the last three months I added about 500 pages and unfortunately almost all of them remain uncrawled by the Yahoo crawler.

It comes regularly to my site but crawls only those 50 legacy pages. (I think, a couple of times it might have grabbed a couple of newer pages.) From my index page, the one that is regularly crawled, I have links to root directories of these new pages but they are left untouched.

The day (March 4?) I submitted my site using addurl (even though there was no need for this) msnbot came and grabbed almost all the pages. A couple of days ago I noticed FAST crawler grabbing pages.

I wonder who is partnering with who and should I be worried about Yahoo not crawling my pages? Thus far I have almost no referrals from Yahoo (and MSN) for these new pages.