Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Message Too Old, No Replies

Yahoo Slurp isn't crawling deeply

Can't get yahoo to deep index my site



3:12 pm on Sep 19, 2005 (gmt 0)

10+ Year Member

I opened a new site on a brand new domain about a month ago. MSN and Google have already indexed many of the pages, but Yahoo is only spidering the home page. I don't have a ton of backlinks but a few and I'm getting more each week. Most of my pages validate XHTML 1.1 and I just recently removed PHP Session IDs completely to make the urls very clean. I don't have robots.txt blocking anything.

Is it normal for Yahoo to take this long before it starts deep crawling, or is there something else going on?


8:57 am on Sep 21, 2005 (gmt 0)

10+ Year Member

This is not normal...
Check you backlinks, it seems Y likes some backlinks and hates others. IMHO


1:02 pm on Sep 21, 2005 (gmt 0)

10+ Year Member

This is normal behaviour that I see on any new sites I work on. Google and MSN will index the entire site quite quickly, but Yahoo often takes up to 6 months to index the whole site.

Generally it will spider the homepage and that'll be it. Even getting deep links from well respected on topic sites doesn't seem to help much.

It'll most likely get indexed immediately before a major Yahoo update.


1:22 pm on Sep 21, 2005 (gmt 0)

10+ Year Member

as for me Y! crawl new sites very well if it like content and backlinks (and domain?). Sometimes it takes up to several months but sometimes it takes 7 days to get indexed


1:55 pm on Sep 21, 2005 (gmt 0)

10+ Year Member

Does anyone here have a complete list of Yahoo's crawlers? We are making some changes to our robots.txt file and I was wondering if "Slurp" is the only known name for all of Yahoo's spiders.


1:04 pm on Sep 22, 2005 (gmt 0)

10+ Year Member

For us, Yahoo likes our home and main pages, but not anythink deeper then a main page. I use to until the last update.

Featured Threads

Hot Threads This Week

Hot Threads This Month