homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
Forum Library, Charter, Moderators: martinibuster

Yahoo Search Engine and Directory Forum

Yahoo Slurp isn't crawling deeply
Can't get yahoo to deep index my site

 3:12 pm on Sep 19, 2005 (gmt 0)

I opened a new site on a brand new domain about a month ago. MSN and Google have already indexed many of the pages, but Yahoo is only spidering the home page. I don't have a ton of backlinks but a few and I'm getting more each week. Most of my pages validate XHTML 1.1 and I just recently removed PHP Session IDs completely to make the urls very clean. I don't have robots.txt blocking anything.

Is it normal for Yahoo to take this long before it starts deep crawling, or is there something else going on?



 8:57 am on Sep 21, 2005 (gmt 0)

This is not normal...
Check you backlinks, it seems Y likes some backlinks and hates others. IMHO


 1:02 pm on Sep 21, 2005 (gmt 0)

This is normal behaviour that I see on any new sites I work on. Google and MSN will index the entire site quite quickly, but Yahoo often takes up to 6 months to index the whole site.

Generally it will spider the homepage and that'll be it. Even getting deep links from well respected on topic sites doesn't seem to help much.

It'll most likely get indexed immediately before a major Yahoo update.


 1:22 pm on Sep 21, 2005 (gmt 0)

as for me Y! crawl new sites very well if it like content and backlinks (and domain?). Sometimes it takes up to several months but sometimes it takes 7 days to get indexed


 1:55 pm on Sep 21, 2005 (gmt 0)

Does anyone here have a complete list of Yahoo's crawlers? We are making some changes to our robots.txt file and I was wondering if "Slurp" is the only known name for all of Yahoo's spiders.


 1:04 pm on Sep 22, 2005 (gmt 0)

For us, Yahoo likes our home and main pages, but not anythink deeper then a main page. I use to until the last update.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Yahoo Search Engine and Directory
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved