What's the Time Frame for Yahoo Slurp Spidering to Inclusion into the Index?

How Long Does it Take to Update Index with New Data?

[sasha]

Over the last several months Yahoo had spidered about 100,000 pages on my site. However the index has remained pretty much the same or increased by maybe 100-200 pages.

How long does it take for Yahoo to include those pages in the index? (I am not talking about the rankings, simply to show indexed pages in domain:mysite.com)...

[martinibuster]

I have seen my sites listed and ranked in a matter of days. Not all of my sites, just some of them. The rest have made it into the index, unranked, within a few days. My understanding is that Yahoo's index is fairly fresh.

[sasha]

so why would they spider the site continuously and then not show the pages in the index? (I am not even talking about rankings here)

[lizardx]

Very very long, I think roughly the same time as the current google sandbox lasts, maybe 6 months, hard to say, yahoo has index capacity issues, just like Google. Fast spidering, like you noted, usually much faster than google, but takes forever to get those spidered urls indexed, into the main index, I think yahoo is only running one index, probably with the same numbers, 4.2 billion pages that google ran with last year, same reason, 2^32 capacity for algo.

Nice to see all the old search engines in the same boat, google hid their problems better, but it's the same problem, yahoo avoids attention because nobody really cares about yahoo, which is silly, good serps are good serps, I'm doing well in yahoo and getting almost the same traffic from yahoo/msn as from google this month.

Still waiting on a site we added a few thousand pages to 4 months ago, not in main yahoo index from what I can tell, in fact yahoo dropped our indexed page count by about 150, but raised us to number one on our single target keyword, go figure. Just not enough room is my guess, simple math, pages have to go before your pages are let in, so no real schedule on that. And sites with a lot of pages get some dumped to make room for new sites, pretty sloppy, wish yahoo could get their stuff working better.

[sasha]

Thank you. This makes more sense to me. From what I have seen they are releasing our URLs that were added about 5 months ago at the rate of about 10-20 URLs per week. Our site has 60,000...

[eyezshine]

New sites get in pretty quick. But if you change URL's on an older site it takes a long time for the old urls to get out and the new urls in.

I have an older site that I changed the URL structure on about 8 months ago and only 11 of the new pages are in the index and I seen Slurp spidering all of the old pages about 3 days ago.

But I have a new site (3 months old) that got indexed right away with 34 pages in the index right now. The new site has over 100,000 pages and slurp continually spiders them deep but hasn't indexed them yet.

So I'd have to agree that 6 months is about right.

I also assume that the more links pointing at you, the deeper your pages get spidered and indexed.

[lizardx]

<< I also assume that the more links pointing at you, the deeper your pages get spidered and indexed. >>

That's not what I've seen, doesn't seem to make any difference at all, the pages get spidered very fast, then it takes a long time for new pages to get in, and old pages get dropped, no particular pattern I can see to that.

I really think that it is not planned behavior, but just no room in their index, that's what it looks like to me anyway, they're not trying to be the best, but I think they are trying to get new sites in, at the expense of number of pages in index of old sites. For some reason they've avoided the intensely passionate scrutiny that google's sandbox type thing gets, but I think it's very much the same problem, with the same cause, only not as consistently implemented as the sandbox.

Time to update from old url's/domain to new one, 6 months, however I don't think that was a timeline thing, I think they just fixed their 301 problem last week finally, so my guess is many sites, irregardless of when the 301's were put in place, were correctly listed.

[mark1615]

We see pages and sites get in quickly. The strange behaviour that we have seen from Yahoo for a year or so is that they will go back to old versions of the index for no apparent reason. We just saw this yesterday. They went back a version of their index that is about 2 weeks old and very different from what they have been showing. Basically its a rollback to what they were showing before the "update." They are even showing old pages of ours in their cache. I would like to say this is unusual (it is definitely strange) but it's not. They just do this sometimes and then they go back to the "current" index. I don't pretend to understand it, I just live with it.

[lizardx]

I'm seeing the same set of serps, no changes, all sites exactly like they were after the update, I'm not complaining :-) ...