Slurp requesting millions, Yahoo shows only a few...?

how can that be..

[flex55]

At the beginning of October 2004 our site was excluded from Y! index. After contacting them, they explained the reason (when a certain error happened in the site, server replied 200 code with and an error page for every request, which led Y! to think there are tons of duplicate pages - the error page).

Yahoo then said that shortly the site will be included back in the index.
Time passed, and indeed, after 4 weeks Slurp started to visit us again.
Since November, Slurp fetched over 5 million pages. 60% of which were in January-
still, a site:domain query returns only 30 pages on yahoo.

Does anyone know what and why? It sounds so weired that Slurp would fetch so many pages and almost none would get into the index.
I'd really appreciate if anyone knows this and what's happening.

[bagira]

funny, we have a similar situation.

- dropped from the YHOO index
- a month later returned with several thousands pages on the index (my guess is that those were the 'old indexed pages')
- after another month our indexed pages dropped to 20-30
- in the last several months YHOO is requesting 5-8 million pages _every_month_, and still shows only 20-30 pages on the index

at first I thought it might take a month or two to YHOO to index all the data, now after 3 months I'm starting to doubt it...

[flex55]

I wonder what that means. I tried to contact them on that feedback form, but got a canned, irrelevant response.
After all, it's so weired- why would they take so many pages if they dont intend to include it in the index.
Maybe they just decided to make us spend money for bandwidth, so we'll be forced to go sitematch :-)

[dataguy]

Yahoo has been hitting my sites so hard in the past week I'm debating whether or not to ban the crawler altogether by redirecting it to a static page.

They have crawled about 100,000 of my pages in the past week, but they only show a few dozen pages of mine in their index. If I don't see something soon, I just don't think it's worth the hassle. (My pages are all dynamically generated, and they take quite a but of processing to generate.)

I guess I'm hoping that they are going to release some big new update that will start sending traffic to my sites, but I'm getting impatient....

[bmsd33]

Same situation. We've only had 16 pages show up despite thousands of pages visited by the Yahoo spider. The one thing I am wondering about is the dynamic nature of the site. Almost all of our links are dynamic, Miva, to be exact. Looking through Yahoo's FAQs, they state the spider prefers static pages for discovery purposes. Looking at the sites that do rank well, I've noticed many have a majority of static html pages indexed, though there are many dynamic pages (maybe the more static pages to jump from, the more dynamic pages?).

Does anyone think there is a connection between low indexed quantities and dynamic pages?

[dataguy]

Does anyone think there is a connection between low indexed quantities and dynamic pages?

Well, if there is, I don't see it. None of my pages use querystrings, they are dynamic, but they end in ".htm" ince the variables are passed in the filename itself.

[flex55]

Maybe Yahoo_Mike or Tim or whoever from Yahoo can shed some light on this?
I really have no idea what to do. This month only, Yahoo took 3,200,000 pages alone- and we're only half way through the month.

I've tried to contact someone from Y!, but only got the canned responses, again. Should I call them? Is there anyone to talk to? It's really frustrating.
There's definitely something wrong going on there. I'm paying lots of money for that bandwidth.. yet 1/1,000,000 of the pages Slurp took gets into the Y! index.

Mike, Tim, help...

[BillyS]

I have the same question about Yahoo. They are by far the most active spider, sucking in my website at least 3 fold each month (the site has 800 page). Yet, Yahoo only shows about 80 pages in their index.

Yahoo has been adding pages steadily, but I actually add pages to the site faster than Yahoo adds pages to their index.

My hope is for a big bang one of these days (or months).

[vabtz]

I have the same issue.

Looking at what is cached by yahoo and what isn't I think the following is applicable to my situation.

- My page sizes were too large ( many over 100k ) it indexed most everything it crawled under 40k

- My dynamic site with mod_rewrite was showing alot of duplicate content for different urls. I changed the robots.txt to stop Slurp from crawling the dynamic looking urls.

- link structure was confusing due to mod_rewrite and I had to many links on my frontpage

Last night I rewrote my pages to be all under 45k for the largest pages and reduced the number of links I presented. I guess we'll see in a month or so.

[bagira]

I don't think this issue is related to page size (our pages are 15-40K). I neither think we have dup content, but maybe I'm missing it...

Moreover,
Isn't it strange that you take 8,000,000 pages and show only 8.
Crawling 8,000,000 pages (multiple 1000? sites that experience this) should cost Yahoo some $$.
So they spend those $$, and just throw the results away?

I'm starting to think they don't do it intentionally (bug on Yahoo?).

[BillyS]

This morning I just checked Yahoo, went from showing 80 pages yesterday to over 500 today. Anyone else seeing an update or is this just an increase to the index for me?

[Dayo_UK]

Yep, Yahoo have increased the page count for my sites too.

Traffic does not seem to follow though......? (Will have to monitor a bit longer).

But Yahoo are doing a much better job nowadays (IMO). Congrats Tim and Yahoo_Mike etc...

Although I am probably talking up an undeserved ban for my site with these type of compliments ;)

[dataguy]

Not really seeing a change for me, maybe it's too early...

[sasha]

too early. It may take 4-6 months for your pages to appear in the index. I just discovered this!

[zeus]

Yes it is incredible that so many sites are only represented with only there index page, but as I have said before is that those sites I have paid for some time ago are those that is only represented with the index page, Im not sure why, but OK im not hurt by the missing yahoo visitors, but I guess others are bigtime.

[steveb]

A few sites show significantly more pages for me, a couple show slightly less, but I still have none of my sites that are under 150 pages fully indexed... where Google has all indexed 100%. Typical example, I put a five page site online a couple weeks ago. All five pages link to all five pages. Yahoo has four in the index. How the heck is that even possible?

[lorenzinho2]

We've had a similar situation - Yahoo spider hammering the heck out of us for the last couple of months, grabbing hundreds of thousands of pages, yet only indexing a small fraction of these.

Yesterday we saw the big jump, at least index wise - the number of pages in the index has tripled.

We're not seeing any increased traffic yet though... maybe that's step three? It would be nice if Yahoo traffic started subsidizing some of the bandwidth costs their spider is causing us.

[BillyS]

As I mentioned earlier, I jumped today from 80 to 550 pages - the site has 800 pages.

I was hoping that my traffic from Yahoo would jump proportionally, but nothing so far. In doing some research, it seems that the concensus is that Yahoo updates SERP anywhere from every 3 days to two weeks.

I monitor search volume daily, so I should be able to see a big change if and when it happens.

[dataguy]

This past week my Yahoo listings jumped an average of about 3 fold. We'll have to see if this means more traffic to my sites or not, haven't seen an increase yet...

[dsz11]

A brand new site of mine (2 weeks old) is being hit hard by Slurp. It is requesting not only the pages I have prepared for crawling for the bots but also significant part of my 4 parameter pages?! I am almost sure it is crawling Mostly such a pages (3:1) ratio.
1.st I have never seen such a big activity from Yahoo for a new site. My experience up to now is that it takes him months before it pays any attention to anything but the index page.
2.nd Since when Yahoo so gladly crawls 4! parameter pages?

Did anybody posting earlier, have already seen any traffic increase as a result of the described Yahoo crawling?

[ichthyous]

My site got hammered in Novemeber when Yahoo dropped all but 300 of my indexed pages. My wonderful Yahoo traffic vanished and has never returned. Slurp is still busy spidering my site day and night, but the number of indexed pages has dropped to 80 now and I get very few visitors from Yahoo. I was wondering if this may be due to the fact that I have "NOARCHIVE" tags on most of my HTML pages and Yahoo may penalize them. Who knows? I have started to spend my time and money on print advetising in order to get away from the dependence on web traffic. It's too treacherous...one day you're up the next you're gone!

[BillyS]

After getting nearly 500 pages in Yahoo about a month ago, those pages are slowly disappearing from the index. The total now stands at 300 even though I have introduced roughly 50 new pages in the past month.

In contrast, it seems like there was an update today and traffic from yahoo continues to increase.