I opened a new site on a brand new domain about a month ago. MSN and Google have already indexed many of the pages, but Yahoo is only spidering the home page. I don't have a ton of backlinks but a few and I'm getting more each week. Most of my pages validate XHTML 1.1 and I just recently removed PHP Session IDs completely to make the urls very clean. I don't have robots.txt blocking anything.
Is it normal for Yahoo to take this long before it starts deep crawling, or is there something else going on?
This is normal behaviour that I see on any new sites I work on. Google and MSN will index the entire site quite quickly, but Yahoo often takes up to 6 months to index the whole site.
Generally it will spider the homepage and that'll be it. Even getting deep links from well respected on topic sites doesn't seem to help much.
It'll most likely get indexed immediately before a major Yahoo update.
as for me Y! crawl new sites very well if it like content and backlinks (and domain?). Sometimes it takes up to several months but sometimes it takes 7 days to get indexed
Does anyone here have a complete list of Yahoo's crawlers? We are making some changes to our robots.txt file and I was wondering if "Slurp" is the only known name for all of Yahoo's spiders.
