Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: martinibuster
Yahoo then said that shortly the site will be included back in the index.
Time passed, and indeed, after 4 weeks Slurp started to visit us again.
Since November, Slurp fetched over 5 million pages. 60% of which were in January-
still, a site:domain query returns only 30 pages on yahoo.
Does anyone know what and why? It sounds so weired that Slurp would fetch so many pages and almost none would get into the index.
I'd really appreciate if anyone knows this and what's happening.
- dropped from the YHOO index
- a month later returned with several thousands pages on the index (my guess is that those were the 'old indexed pages')
- after another month our indexed pages dropped to 20-30
- in the last several months YHOO is requesting 5-8 million pages _every_month_, and still shows only 20-30 pages on the index
at first I thought it might take a month or two to YHOO to index all the data, now after 3 months I'm starting to doubt it...
They have crawled about 100,000 of my pages in the past week, but they only show a few dozen pages of mine in their index. If I don't see something soon, I just don't think it's worth the hassle. (My pages are all dynamically generated, and they take quite a but of processing to generate.)
I guess I'm hoping that they are going to release some big new update that will start sending traffic to my sites, but I'm getting impatient....
Does anyone think there is a connection between low indexed quantities and dynamic pages?
I've tried to contact someone from Y!, but only got the canned responses, again. Should I call them? Is there anyone to talk to? It's really frustrating.
There's definitely something wrong going on there. I'm paying lots of money for that bandwidth.. yet 1/1,000,000 of the pages Slurp took gets into the Y! index.
Mike, Tim, help...
Yahoo has been adding pages steadily, but I actually add pages to the site faster than Yahoo adds pages to their index.
My hope is for a big bang one of these days (or months).
Looking at what is cached by yahoo and what isn't I think the following is applicable to my situation.
- My page sizes were too large ( many over 100k ) it indexed most everything it crawled under 40k
- My dynamic site with mod_rewrite was showing alot of duplicate content for different urls. I changed the robots.txt to stop Slurp from crawling the dynamic looking urls.
- link structure was confusing due to mod_rewrite and I had to many links on my frontpage
Last night I rewrote my pages to be all under 45k for the largest pages and reduced the number of links I presented. I guess we'll see in a month or so.
Isn't it strange that you take 8,000,000 pages and show only 8.
Crawling 8,000,000 pages (multiple 1000? sites that experience this) should cost Yahoo some $$.
So they spend those $$, and just throw the results away?
I'm starting to think they don't do it intentionally (bug on Yahoo?).
Traffic does not seem to follow though......? (Will have to monitor a bit longer).
But Yahoo are doing a much better job nowadays (IMO). Congrats Tim and Yahoo_Mike etc...
Although I am probably talking up an undeserved ban for my site with these type of compliments ;)
Yesterday we saw the big jump, at least index wise - the number of pages in the index has tripled.
We're not seeing any increased traffic yet though... maybe that's step three? It would be nice if Yahoo traffic started subsidizing some of the bandwidth costs their spider is causing us.
I was hoping that my traffic from Yahoo would jump proportionally, but nothing so far. In doing some research, it seems that the concensus is that Yahoo updates SERP anywhere from every 3 days to two weeks.
I monitor search volume daily, so I should be able to see a big change if and when it happens.
Did anybody posting earlier, have already seen any traffic increase as a result of the described Yahoo crawling?
In contrast, it seems like there was an update today and traffic from yahoo continues to increase.