Welcome to WebmasterWorld Guest from

Forum Moderators: martinibuster

Message Too Old, No Replies

At the rate Yahoo indexes new pages

they'll never catch up!



6:15 pm on Mar 31, 2004 (gmt 0)

10+ Year Member

While I think Yahoos! SERPS are definitely competitive with Googles, they are definitely WAY behind in their ability to add new pages to their relatively (to Google at least) small index.

We operate several large sites and Google has indexed well over 30x what Yahoo! has. And at the rate they're adding new pages, they'll never catch up.

So, Tim, when are you guys rolling out a more competitive indexing bot or was all the talk at PubCon about Yahoo!'s commitment to indexing the ENTIRE web for free just talk?

Chicken Juggler

6:20 pm on Mar 31, 2004 (gmt 0)

Google updates every day almost. I can get new pages in 24hrs. It would not be good for site match if yahoo did the same. They would lose money if they did what Google does. Many of us would not have signed up for sitematch if Yahoo updated like Google. I can't wait 3 months or even a month to have my sites update. That is why I paid. Yahoo is a PPC engine now.


6:23 pm on Mar 31, 2004 (gmt 0)

10+ Year Member

>> Yahoo is a PPC engine now

I'd disagree, for many Yahoo produces good results it is only a small percentage that would need to use the PPC.


6:41 pm on Mar 31, 2004 (gmt 0)

10+ Year Member


That is a really good observation.

Google picks up new changes within a day usually.

I use a redirect on a domain because people mispell my domain name. Yahoo has this result on top when you search for my site. I changed the page to have a link and no content. This was 2 weeks ago and Yahoo hasnt picked it up.

Something like this depreciates the value of search results on Yahoo because users are taken to a page that no longer contains the content that Yahoo is showing it to have.


6:49 pm on Mar 31, 2004 (gmt 0)

10+ Year Member

Yahoo has implemented a pay for listing service which gives them no incentive to improve free spidering performance. I believe they have no plans to truely try to index the web in a competitive manor. That being said, I would love for them to prove me wrong.

Cabos, I have a similar situation, just that mines been like that for more than a couple weeks.
1. I made the mistake of registering a new domain name and pointing it to another site of mine using a CNAME. No external sites to my knowledge ever linked to this (new) name.
2. My entire main site was indexed my Ink using the temporary cname intended for another site.
3. My main site is nowhere to be found under its proper domain in any of the ink engines.
4. I notice the problem almost a year ago now and implemented a perminent http header redirect.
5. Almost a year later my main site can only be found by ink search engines with the wrong name. I don't want to turn off the redirect since I get traffic from it.
6. This problem is now in the new Yahoo engine.
Google has never once indexed the site by the wrong name. I can't be the only one with problems like this. I think it's a very poor quality spider and engine.

[edited by: peterdaly at 7:03 pm (utc) on Mar. 31, 2004]


6:58 pm on Mar 31, 2004 (gmt 0)

10+ Year Member

In order to remain competitive they will need to keep their results fresh.


7:23 pm on Mar 31, 2004 (gmt 0)

10+ Year Member

The problem with site match is that one of the main features they are using to sell it is something both content providers and searches expect - fresh content.

Google has raised the bar by saying to searches, you will find really fresh content and by letting content providers know that new content will show up within a couple of days.

Yahoo is trying to sell the freshness feature as a benefit. Its almost like they feel they have monopolistic power and can force users to pay for freshness. The problem for them is they arent the only search engine in town.

As much as they are convinced that freshness can be sold, it's a feature that is a standard offering of their major competitor.

The marketplace will rule. In the short term Yahoo will see a bump up in site match revenues, as advertisers experiment, and new sites try it due to Yahoo's aggresive marketing, though over time as consumer knowledge increases, the revenues from site match will drop off.


7:27 pm on Mar 31, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

>> no incentive to improve free spidering performance.

GG said along these lines a year ago.

I think the part where we differed is on pay-for-inclusion. We've never done it because we thought it was better for users not to bring money into our editorial results; there's the danger that you create an incentive not to crawl as well so that webmasters have to pay to get crawled.

[webmasterworld.com...] (emphasis mine.)

Chicken Juggler

7:35 pm on Mar 31, 2004 (gmt 0)

Well you know they are looking at the bottom line. Yahoo is guaranted a certain number of visitors just because they have been around a long time and have the portal. The amount of money they make from only allowing paid fresh results is more than they lose from people going to Google. They will keep tweaking it so they make more and more money. Money goes down a little they will tweak it some more. That is how you run a business. You do what you can get away with.


7:56 pm on Mar 31, 2004 (gmt 0)

10+ Year Member


Tweaking may work for them in the short term.

At the moment they have the benefit of a large user base. The problem is that we are all strapped for time and will go to the search engine that produces the best product/results.

The herd mentality and "buzz" about what search engine is best will dictate. Negative buzz can kill a product fast.

In a short period of time, Yahoo has created plenty of buzz. Question though whether this buzz is positive or negative?

Chicken Juggler

8:00 pm on Mar 31, 2004 (gmt 0)

I'm sure they have their finger to the wind. Coke recovered from New Coke. Yahoo could recover from something like that. Plus it is not run by one person is it? People make stupid decisions all the time to benefit themselves and hurt their own company. Somebody might just make a killing take his money and run and leave yahoo to pick up. It is not uncommon for companies to do really stupid things on purpose because the people currently running the business won't have to deal with it.


8:06 pm on Mar 31, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

The herd mentality and "buzz" about what search engine is best will dictate. Negative buzz can kill a product fast.

Particularly if your main competitor is busy explaining why they think the strategy you adopted is flawed. While G's management is not directly criticizing Y's approach, they have given a number of interviews explaining their negative views of PFI and why they don't think they should go through that route. G has certainly been busy distancing themselves from PFI, and explaining why the approach they are taking is the high road to search.


1:59 pm on Apr 2, 2004 (gmt 0)

10+ Year Member

Yesterday Google spidered 580+ pages on our main site while Inktomi crawled a massive 8 pages. I really doubt that Yahoo! is serious about indexing the entire web in our lifetimes.

Tim, care to state otherwise?


Featured Threads

Hot Threads This Week

Hot Threads This Month