Forum Moderators: open

Message Too Old, No Replies

Yahoo Slurp Spidering Erratic?

Yahoo Slurp Spidering Erratic?

         

energylevel

1:59 am on Dec 30, 2004 (gmt 0)

10+ Year Member



Has anyone noted a pattern or particular requirements for getting pages spidered and included in the Yahoo search results, I've noticed some things but I'm not sure whether they are the norm:

Yahoo only listing the homepage of a new site (3 months old now) at the moment? I've seen this before.

About half of the pages on a small site (40 pages) are listed the rest are not listed at all, there seems to be no pattern or reason for excluding half the pages as far as I can see?

lizardx

10:10 pm on Jan 3, 2005 (gmt 0)

10+ Year Member



Yes, I've seen this, there is very little obvious patterning of Yahoo's behavior, it's got a definite random tone to it, however, I've finally realized that there is possibly an underlying logic, keeping in mind that around last year the number of webpages in the world exceeded what I think the capacity of search engines was, both yahoo and google.

So Yahoo to me looks like it's doing something like just getting rid of some percent of some sites pages, this makes room for new sites, but the process is slow, and very unpredictable, but does seem to come in waves, suddenly one day many people notice that number of pages indexed has dropped dramatically.

this is a cheap way to maintain the appearance of freshness, by dumping large sites' pages, you make room for newer sites, but many sites are not fully indexed.

Others are, but it takes a long time. Being an authority type site seems to help, but that could also be random.