Welcome to WebmasterWorld Guest from 50.17.117.221

Forum Moderators: martinibuster

Message Too Old, No Replies

Yahoo Slurp isn't crawling deeply

Can't get yahoo to deep index my site

     
3:12 pm on Sep 19, 2005 (gmt 0)

New User

10+ Year Member

joined:May 15, 2005
posts:24
votes: 0


I opened a new site on a brand new domain about a month ago. MSN and Google have already indexed many of the pages, but Yahoo is only spidering the home page. I don't have a ton of backlinks but a few and I'm getting more each week. Most of my pages validate XHTML 1.1 and I just recently removed PHP Session IDs completely to make the urls very clean. I don't have robots.txt blocking anything.

Is it normal for Yahoo to take this long before it starts deep crawling, or is there something else going on?

8:57 am on Sept 21, 2005 (gmt 0)

New User

10+ Year Member

joined:Sept 18, 2005
posts:3
votes: 0


This is not normal...
Check you backlinks, it seems Y likes some backlinks and hates others. IMHO
1:02 pm on Sept 21, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 7, 2005
posts:86
votes: 1


This is normal behaviour that I see on any new sites I work on. Google and MSN will index the entire site quite quickly, but Yahoo often takes up to 6 months to index the whole site.

Generally it will spider the homepage and that'll be it. Even getting deep links from well respected on topic sites doesn't seem to help much.

It'll most likely get indexed immediately before a major Yahoo update.

1:22 pm on Sept 21, 2005 (gmt 0)

New User

10+ Year Member

joined:Sept 18, 2005
posts:3
votes: 0


as for me Y! crawl new sites very well if it like content and backlinks (and domain?). Sometimes it takes up to several months but sometimes it takes 7 days to get indexed
1:55 pm on Sept 21, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Nov 10, 2003
posts:238
votes: 0


Does anyone here have a complete list of Yahoo's crawlers? We are making some changes to our robots.txt file and I was wondering if "Slurp" is the only known name for all of Yahoo's spiders.
1:04 pm on Sept 22, 2005 (gmt 0)

Full Member

10+ Year Member

joined:Apr 7, 2003
posts:257
votes: 0


For us, Yahoo likes our home and main pages, but not anythink deeper then a main page. I use to until the last update.