Forum Moderators: open

Message Too Old, No Replies

Inktomi Core DB Update

A major update underway?

         

przero2

4:23 pm on Aug 8, 2002 (gmt 0)



Did Inktomi do a major update ... I am seeing major differences in referrals from MSN and indications are pointing to a huge Inktomi core DB update? Any one else see this?

duhboy

3:57 pm on Aug 9, 2002 (gmt 0)

10+ Year Member



Hello all
I'm a newbie. Nothing wrong there that time won't fix I suppose.
Could someone please provide me with the kind of spider identification information I could be looking for in my log files.
Specifically for Inktomi and Goodle.
I am always grateful.
Thanks, Duhboy

incywincy

3:59 pm on Aug 9, 2002 (gmt 0)

10+ Year Member



hi duhboy,

look for the follwoing user agents in your access logs:

slurp = inktomi
googlebot = google

good luck!

Robert Charlton

4:14 pm on Aug 9, 2002 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Well, I'm also seeing evidence of deep spidering and new pages shown for the same site where (as I posted above) the home page results looked over a year old.

Looking at the home page title more clearly, there's obviously been some sort of database glitch... The title is truncated and pasted together twice, definitely not what's on the site... nor is the description close to what's there.

Otherwise, the rest of the pages look good and are a welcome addition. Wonder what the home page glitch is about.

skibum

4:29 pm on Aug 9, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If a page is listed 2x in Open Directory, sometimes Ink will show a combination of the two listings all neatly combined into one that makes for something that looks really bizarre.

GoInkGo

4:40 pm on Aug 9, 2002 (gmt 0)

10+ Year Member



There is something wrong with this update, as of august 9. It's strange... MSN shows different results than pure Ink. I see a lot of irelevant websites/pages, ugly sites and japanese pages for a search query.

These pages are free ones.

UPDATE:

After writing the message above, I did another test at MSN and results are back to normal (I think)... all sites mentionned above are not there anymore.

nell

4:44 pm on Aug 9, 2002 (gmt 0)

10+ Year Member



>all sites mentioned above are not there anymore.

Remove your pair of Inktomi glasses and look again.

They have finally acknowledged that the direction they were going was taking them nowhere but down. They have gone back to the last situation where life was good to them and have started over from there.

They have put up every page in their database that's still alive and ranked them, to what they believe, is according to some relevance. This, for them, is square #1.

The PPI listings will be spidered every 48hrs, the others every ??.
The SEO now needs to revise any low ranking pages so that when they're spidered in 48 hrs. they push down the free listings. Now your PPI pages are competing with all pages not just the other PPI pages. However, after the free listings get their infrequent spidering, you could find those pages back on top of yours again. Then repeating your process to SEO them back up.

The problem now is that the current results are full of old spam pages. To SEO your PPI pages back up to the top means spamming your own pages too. Unless at some point INK cleans out the spam pages their search results will be nothing more than a meal of spam sphagetti.

Will INK filter the spam out? Probably. If/and or when they do at some later date and, in the meanwhile, you have had to spam your PPI pages to get back to the top, you'll be booted out too. It means you take the risk of spamming or resign yourself to where you now are.

Personally, I don't think they give a rat's about the PPI customers anyway. No more than LS cared about their one-time-fee paid customers. They've both milked their respective cows dry. PPC is the new wave, not PPI. To stay alive they need money. Watch for "new product releases" that are PPC oriented. In the end, the wallet will rule.

GoInkGo

5:13 pm on Aug 9, 2002 (gmt 0)

10+ Year Member



The problem I saw some minutes ago just came back at MSN :( Hope that it's just a glitch and everything will come back as normal soon.

There are some pages added in the new update that no longer exist (these page are dated of 1999!)

Paid pages are below the free pages (1-28: free - Paid ones start at #29)

Something wrong with the update, definitively.

By the way, I see the problem with msn.ca ; msn.com seems to be Ok... but I'm not sure.

GoInkGo

5:15 pm on Aug 9, 2002 (gmt 0)

10+ Year Member



I don't like this database update :(

Marcia

5:32 pm on Aug 9, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



<side note>
duhboy, check the Search Engine Spider Identification [webmasterworld.com] forum for User Agents and IP numbers.
</side note>

duhboy

5:36 pm on Aug 9, 2002 (gmt 0)

10+ Year Member



Thanks Marcia.
Will do. Duhboy.

kris

6:39 pm on Aug 9, 2002 (gmt 0)

10+ Year Member



Haven't heard from Inktomiguy since:

Glad to see you noticed our major cluster release. We're extremely proud of it for a number of reasons. It's the largest, most relevant and freshest by far.

Let us know what you think!

Well we are letting you know, can you explain the findings here?

Even GoInkGo doesn't like the update?!?!

Everyman

7:24 pm on Aug 9, 2002 (gmt 0)



Now your PPI pages are competing with all pages not just the other PPI pages. However, after the free listings get their infrequent spidering, you could find those pages back on top of yours again. Then repeating your process to SEO them back up.

And maybe, just maybe, the backfill is why most searchers use search engines in the first place. If they want ads in the morning, ads in the evening, ads all day long, they can junk their computer and go back to watching TeeVee.

It seems to me that Inktomi realized that a pure paid approach without backfill merely leaves the entire search field to the competition. Google found themselves in a position to claim more search integrity. Then Google started attracting searchers Big Time, because most objective observers discovered that Google was right (that was two years ago; things may be changing by now). Inktomi skipped this step, and now they're trying to pick up the pieces by using deep crawls and ranking algorithms on unpaid content from the web.

There are only a few outfits that can do coherent deep crawling and decent ranking too. Google, Alltheweb, and Inktomi come to mind. If you can do it, then you can compete. If you can't do it, you muddle about (e.g., AltaVista). Most searchers are not searching so that they can buy your widgets.

From the point of view of the search engine, if you can do it but you don't do it because of a mindset that is more focused on some particular revenue model, then you end up losing to the competition and you also end up without even an excuse.

Paying for crawl inclusion is essentially in the same category of paid ads. The FTC didn't address it on this level, but I don't see a whole lot of difference. If you pay to get included, it's because if you didn't pay, the searcher wouldn't be looking at your link after doing an Inktomi search. Your link appears in the SERPs because you paid.

How is this different than what the FTC was talking about? Why shouldn't this sort of inclusion be conspicuously tagged in the SERPs as "paid" or "sponsored"?

thunderpaste

8:04 pm on Aug 9, 2002 (gmt 0)

10+ Year Member



After this "update" my rankings at MSN stayed pretty solid but out of the 19:( sites above me 6 are duplicate sites under different domains and the rest are doorway pages using keyword rich sub-domains. Not very impressive to me.

crystalkey

11:10 pm on Aug 9, 2002 (gmt 0)

10+ Year Member



>Paid pages are below the free pages (1-28: free - Paid ones start at #29)
I don't understand why Ink tomi need to lower PFI websites.
PFI is nothing but guarantee the inclusion of website and shouldn't affect ranking.
Everyone perticipated PFI maybe upset.
I don't see the point.

GoogleGuy

11:31 pm on Aug 9, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



kris: you gotta give Inktomiguy a chance. If it's someone stopping by on their free time, it might take a while to reply. Give the benefit of the doubt.

Everyman: I thought the FTC did mention making pay-for-inclusion clear as well? Don't have the doc in front of me though. I didn't pay as much attention to that since Google doesn't do PFI. :)

Everyman

12:48 am on Aug 10, 2002 (gmt 0)



GoogleGuy, I think you're right about the FTC. Looking over the clippings I saved, it appears that the FTC was concerned in a very generic sense about any payments that influence search results.

But I've spent several hours studying the difference between Inktomi and Google with respect to the sensational statistics I've been able to compile over the last two days. There's something as sinister as payment for placement going on, and it goes by the name of PageRank.

Both Inktomi and Google have something like 55,000 to 90,000 pages of mine in their database. It depends on the technique used to count the pages.

I track referrals on two levels. Level One are the fewer than 1,000 pages near the surface of my PR 7 site. These are book reviews and essays, and some special topic pages. From the PR 7 passed down from my home page, and from external linking, most of these surface pages have a PR of 4 or 5. Google referrals have been steady at anywhere from 32 to 42 percent of all referrals for these pages. No one even comes close to Google. The closest is around 5 or 10 percent.

Then there are the deep pages, Level Two. Too many thousands of pages means that most of these end up showing PR 0 on the toolbar. Perhaps about 20 percent are lucky enough to show a PR 1. There's no penalty involved; it's just that I ran out of PR to pass around, and there's almost no external linking to these pages.

Starting yesterday, I had a real slap in the face about the importance of the low PR on these internal pages. Two days ago, Google was steady at about 48 percent of all referrals for these deep pages. Google had Inktomi beat by a 4-to-1 margin. Then starting yesterday, Inktomi had Google beat by about a 2.2-to-1 margin. Nothing changed at Google. And we're talking thousands of hits per day, so it's not like I'm running stats on a tiny sample. At the same time that this dramatic shift occurred on the deep pages, there was NO change in the advantage of Google over Inktomi on the surface pages.

Here's my analysis: In both Google and Inktomi, having the name in the title is very important. But the difference is that while good on-page techniques are helpful with both engines, the fact is that on Google, whatever I achieve with good design gets pushed back down by a low PR for that page. This doesn't happen in Inktomi.

So here's the bottom line: PR works against the little guy. Little guys are not allowed to have big databases. Period. Because even when they manage to get the whole thing crawled (something I'm finally beginning to achieve with Google, after nearly two years of effort, since the crawl is also PR-driven), their internal PR is so crummy that it amounts to discrimination.

Only big guys can have big sites that really work well in Google. Little guys should stick to sites with pictures of their cat Fluffy and their dog Spot.

This is such a fundamental flaw with PageRank, that it amounts to Google's Original Sin. PageRank reinforces the powers that be. PageRank discourages power-structure research. PageRank challenges us to overthrow it.

GoInkGo

1:07 am on Aug 10, 2002 (gmt 0)

10+ Year Member



PFI is the best thing a search engine/provider can offer to webmasters :) There are so many advantages ! Both for the SE/provider and for the webmasters. I don't understand why you, Google, didn't started a PFI program :)

I'm in love with the Inktomi's PFI program (and with the company overall :) ) and I hope that it will last for years ! :)

But the subject of this thread is not about the PFI programs but about the Inktomi's database update...

I think that this 'update' must be re-updated... :)

So... we are waiting for your input, Inktomi ;)

Go Inktomi GO ! (Thumb Up) :)

nowhere

2:38 am on Aug 10, 2002 (gmt 0)

10+ Year Member



Look on the bright side; it may be back to pumping out tripod pages by the hundreds each day and getting them in for free.

przero2

2:48 am on Aug 10, 2002 (gmt 0)



I agree with GoInkGo that this must be re-updated as I still see a huge preference in results to PFI pages. Of the searches I examined, a lot of them come up ahead for no reason except that they bought their positions. I believe PFI has run out of its time and this recent update is perhaps one of the "early" indicators of this. Inktomi should go the Google way in true spirit. Provide nothing but unbiased results based on relevance/algo and then sponsored listings similar to Adword Select for advertisers. If Inktomi can do that, our ad dollars are going to follow which BTW could be many times what it would amount to if we had submitted all the home pages to Ink's current PFI model!

Doofus

3:47 am on Aug 10, 2002 (gmt 0)



Of the searches I examined, a lot of them come up ahead for no reason except that they bought their positions.

How do you tell the difference between unpaid and paid inclusion? Are they marked in some way? If not, is it Inktomi's responsibility, or the responsibility of the interface provider (such as Microsoft) to mark them, if indeed it is determined that marking is required to comply with FTC guidelines?

GoInkGo

4:01 am on Aug 10, 2002 (gmt 0)

10+ Year Member



Doofus... If you see:

redirect-west.inktomi.com

When you move your mouse over the titles of the sites, these pages are paid ones.

P.S. Me, I saw a preference to FREE pages... not paid ones. Like I told in a previous post, 1-28 are free, paid pages start at #29 and free pages continue to be there; but the first paid one is at #29. I'm sure (at 99.8% :) )that link popularity don't play any role for the sites located in the 1-28 range.

mayor

12:25 pm on Aug 10, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Anyone else see a big drop in their PFI traffic as measured by the Positiontech stats on daily click-throughs?

nell

3:05 pm on Aug 10, 2002 (gmt 0)

10+ Year Member



>Anyone else see a big drop in their PFI traffic

Not really, a large number of our PFI pages were optimized strong enough to maintain their respective 1/2/3 positions despite the "downdate". In fact, a lot of competition has been pushed way down or even off the first page so our conversions are now higher.

Inktomi

4:08 pm on Aug 10, 2002 (gmt 0)

10+ Year Member



Hi all,

Thanks for all of the feedback. Given the slew of both positive and negative thoughts, I wanted to reinforce the following points:

(1) Our PFI program does *not* impact ranking and never has. Our first priority is quality results. There seems to be a lot of anecdotal evidence presented here on both sides of that debate. But paid inclusion customers are only ensured that their content is crawled and refreshed every 48 hours. The content acquired via our paid inclusion programs enhances the search experience by ensuring that the index includes high-quality deep Web content and that all content is up-to-date.

(2) GoogleGuy is right - the FTC letter does mention PFI. However, their recommendations are that search engines clearly disclose when some of their results are PFI (which we do) and that they highlight any PFI results which impact ranking (which does not apply to us). Our legal folks have examined the FTC document and we believe that we are in complete compliance.

Thanks,
Inktomi.

Everyman

5:23 pm on Aug 10, 2002 (gmt 0)



...search engines clearly disclose when some of their results are PFI (which we do) and that they highlight any PFI results which impact ranking (which does not apply to us). Our legal folks have examined the FTC document and we believe that we are in complete compliance.

Thank you, Inktomi. I'm comfortable with your interpretation of the FTC guidelines. After all, paid inclusion on the crawl is no worse than PageRank inclusion on the crawl. Many large sites with low PR have trouble getting complete coverage from Googlebot. While your implementation of PFI would not be an option for such sites on a per-page basis, it's the philosophy that interests me. Philosophically speaking, that fact that high PageRank sites get crawled more thoroughly than low PageRank sites, is neither more nor less discriminatory than the fact that some webmasters may feel the need to pay Inktomi for inclusion.

The key thing is that the payment to Inktomi doesn't affect ranking. Of course, we all know that PageRank has a tremendous affect on ranking in Google, but that's another issue for another forum. I think Inktomi actually comes comes out cleaner in the comparision, once you see PageRank as a form of currency. Rich people have more money, just like big business has more PageRank.

nell

5:46 pm on Aug 10, 2002 (gmt 0)

10+ Year Member



>highlight any PFI results which impact ranking (which does not apply to us)

To comply with the FTC they had 2 choices.
(1) highlight the PFI listings
(2) neutralize the PFI listings and make it a free-for-all

Choice (2) results in many PFI listings to drop so low that they're worthless. That forces 2 choices on the PPI customers.
(1) SEO every low ranked PPI page back to where it was (and again and again whenever they decide to re-shuffle the search results)
(2) upgrade from the nickle/dime PPI program into a "sponsor" program

As only deep pocket customers will make choice (2), expect a new "PPC oriented product" released to "help the rankings for the budget-minded".

All this praise to them about providing "pure unadulterated search results" is bull. Do you think INK did this because they all woke up one morning and saw "Jesus" ? Or perhaps something do with making $$$ and keeping themself in busine$$ ? This FTC ruling was used to make beneficial change$.

Coincidental that Monsieur "Inktomi" came out of the closet a few days before this big change? This is an INK public relations man and nothing more. Where was he for the last 6 months when we had problems that required answers of substance? The same place he'll be whenever we need answers of substance.

Jack_Straw

8:25 pm on Aug 10, 2002 (gmt 0)

10+ Year Member



Others on this forum have said that PFI is a failed model and I couldn't agree more. As, Inktomi has learned (and should have known from the beginning - it was clear to us), the model is self-contradictory and doomed to failure.

Inktomi once enjoyed the position as the premier search engine on the web. True, in an increasingly sophisticated environment, they became challenged with inferior technology (content based ranking) that undermined the value of their product (their results became ridden with spam) and left them weak in the face of competition based on more sophisticated technology (link analysis ranking – Google). But those issues could have been addressed with some good engineers, some money and an overhaul of their technology.

It is no coincidence that Inktomi’s decline in their share of the search market coincided with their move to PFI. People use search engines to get information, not advertisements. When they cut back on their crawling, they cut back on their relevance and value to a market seeking information relevant to their searches.

The history of advertising has been riddled with attempts of advertisers to promote their ads as information. It was in the late 70’s or early 80’s, as I remember, that this practice blossomed in the magazine industry. I remember “Readers Digest”, as a particularly blatant practitioner, but for a couple of years it was everywhere. You are reading what you think is an article in Time or Newsweek when you suddenly realize that you have been duped into reading an advertisement. It was outrageous and consumer advocates were able to force laws that stopped the practice and required that ads be clearly labeled and separated from information.

So, Inktomi wants to, once again, become a search engine because their customers (MSN and Yahoo) need search results (information) to put under their advertisements. The position of MSN, right now, is particularly enlightening: They have MSN ads, Overture ads, Looksmart ads and Inktomi ads. As people figure out that that is all they have, they loose market share (to Google mostly). MSN and Yahoo need to offer ads, but they need to offer information too or they will loose their audience. Inktomi’s only chance for survival is to fill that need. So they return to being a search engine.

Inktomi can only be a source of one (not both advertisements and information), and they have changed their minds and discovered that they must provide information. A good move for them, I think, if it isn’t too late.

To the extent they attempted to be both in the past, they have sown confusion, caused animosity and exposed themselves, once again, as the unscrupulous money-seeking corporation they are.

Why did they feel compelled to stop spidering? Because, sites could be spidered and included for free, why would they participate in PFI?

Why did they feel compelled to unscrupulously drop all other pages from their index for sites that submitted a page to the PFI program? To force those sites to pay for their advertising.

If Inktomi were clearly, consistently and unambiguously an advertising medium, nobody would resent their refusal to advertise for free.

From the other side, consider the clear advertiser, like GoInkGo. Of course he now resents his loss of market share and that all those free listings beat out his paid listing. He paid to his advertisements – why should all those "freeloaders" be able to rank above him?

Hey, GoInkGo, maybe you should give it up. Change your name to GoLSGo, or GoOvertureGo (Once it could have been GoGoGo :) ) or something. Both are firmly committed to advertisements and are unlikely to change their minds on you.

Also, consider the PFI listing that suddenly stopped showing (as often happened) for some reason. The lister paid Inktomi for the advertisement and they suppressed it without notification. Not fair! You paid for an ad and they didn’t show it.

I think the term PFI, in the current context of what Inktomi is trying to do, is misleading and will continue to cause confusion and resentment.

In some cases, it makes sense to pay Inktomi for their program. But, you are not paying for inclusion. It is easy enough to be included without paying because Inktomi has returned to full spidering.

What are you paying for? You are paying for frequent spiderings of your listings. Inktomi now offers a Pay for Fresh service. If you have frequently changing content, it is very valuable to be able to get your listings updated every 48 hours. Also, if you are in SEO, it can be valuable to get frequent updates in order to test out your SEO techniques.

This is a PFF (Pay for Fresh) service, not PFI. A valuable service! InktomiGuy, just a few posts up, said that this is their value proposition and I agree. But stop calling it PFI. That causes confusion and misunderstanding. The only people who will pay for PFI in a database supported by an aggressively crawling spider are newbies who have not figured out how to get crawled for free. Anybody who pays for inclusion will resent others being listed for free, especially when those listing rank above them.

przero2

10:00 pm on Aug 10, 2002 (gmt 0)



Everyman, the key difference between PageRank and PFI is the "PAY" part of it. PageRank cannot be bought (at least directly from its maintainer Google:)). Each SE indexes per their algo and PageRank happens to be one (big) factor for Google's algo. But let us admit that Google does not charge for the PR. If you disagree, I am curious to know how PR could be bought from the source Google, Inc.

przero2

10:07 pm on Aug 10, 2002 (gmt 0)



jack_straw, a good post!. I agree with you on the confusion part of PFI when all it does is pay for fresh (frequent spidering). I think the way this "confusion" gets resolved over (a pure speculation on my part) is Inktomi dropping the ball on PFI and go the way of AdwordsSelect!. Then no FTC worries, no confusion and all is a fair play with Ink competing head on with other SEs for relevance. A close 2nd or 3rd in the overall search relavance market (exclude the PPCs and other pay for something types) for Ink may be all it needs to regain some of the lost lustre!

Everyman

10:30 pm on Aug 10, 2002 (gmt 0)



But let us admit that Google does not charge for the PR. If you disagree, I am curious to know how PR could be bought from the source Google, Inc.

I disagree. Votes cannot be legally bought either, but Washington is swarming with lobbyists who get laws passed for special interests. There are many, many subtle ways to achieve through the back door, what cannot be done through the front door.

A direct cash exchange is but one form of currency transfer. What we're really interested in is transfer of, and recognition of, social power. Cash is one representation of social power. PageRank is another. On the Web, PageRank is pure power, thanks to Google's near-monopoly.

It's often said that one needs to have money in order to make money. So it is with PageRank. The more PageRank you have, the richer you are. The richer you are, the better equipped you are to become even richer. As Google becomes more of a monopoly on the Web, PageRank is the same as money.

On another forum, there's a lively discussion concerning the viability of putting a text-box ad on your site and charging for the link according to the PageRank that you enjoy. Just because Google doesn't charge, doesn't mean that PageRank is not a form of currency. Google is like the Federal Reserve System of the Web. They control the PageRank supply. If they don't like you, they can take you out of the game. SEOs often talk about how they'd be willing to pay to get back in.

You have to see money as a symbol, or expression of, social power, and then you will understand PageRank. Google doesn't charge directly, but many of us pay a price for Google's concentration of power, and the quirks of their algos.

This 107 message thread spans 4 pages: 107