Forum Moderators: open
Why couldn't there be some account system, like AdWords, where webmasters can pay for an immediate deep crawl to get new content indexed? That way, us low PR sites would have a chance of increasing our rank as we add content. It could even be limited, like no more than two deep crawls per month per domain.
Notice that I'm not asking to buy a high ranking; I'm just asking for the chance to get some added material included in the index. I think this is a very feasible idea and one that can only improve the Google search results.
Deepcrawl as we know it may not exist anymore.
Patience is the key.
There are quite a few people who believe a PFI option is only a matter of time. Just a speculation though.
We'll see how Google's indexing behaviour will change when finally the dust of this non update period at Google has settled.
In all fairness it should be noted that Google has been the best and deepest indexing engine over the last two years or so.
Lately Google has been slacking, while ATW and Inktomi, even AV have been getting more active.
Several search engines have crawled my site within the past week or so. Inktomi's was by far the largest crawl; Google's was the smallest of all identified crawls.
Crawling is only half the battle. For a long time, AltaVista's crawler was sucking up my pages like a vacuum cleaner, but the AV index wasn't updated for months. Google has always been the champ at adding pages (since the days of Infoseek's "instant idexing" feature, anyway). Thanks to the freshbot, I've had new pages appear in the Google index within a day or two. Still, it's good to hear that other search engines are finally trying to catch up.
I begin to wonder if there is room for a completely new PPI engine a little like the print Yellow Pages...
There are millions of websites that will take your $$ for inclusion in their 4 page views per day "directory".
The problem is the chicken and egg scenario. Yes - a good PPI indexed SE probably has a place now but how do they get the traffic up front to make paying for inclusion worthwhile?
If the answer is "make it a very low cost to begin with", then unfortunately you make it worthwhile for the Internet noise to register and don't gain anything.
Google however, already has the traffic...
I happen to agree with Google. It is not pure extra revenue, it would cost them a lot of money in the long run in lost ad revenues.
Even if you are not guaranteed position, it still taints the results. You are gaining an advantage in the number of pages that are being indexed in comparison to a similar site that might even have more links coming in.
What will happen is that commercial sites will flood google with any sort of garbage page knowing that they can get into the index. While more important informational sites will get a normal crawl, or possibly even less of a crawl because the bots are busy with the "paying" cuatomers.
The balance of the index will tilt towards the paying commercial sites.
Now other than the actual taining of the index, there is the even larger problem of the lost trust in the results by the users. Going PFI was the beginning of the end for many of the other search engines.
If google loses just 10% of their users due to lost trust, that would be $75 million. How much do you think google would be able to make on PFI without tossing out their free crawl? I don't think they would even break even.
Then imagine the complaints when google decides to drop these pages, or donwgrades them when they change the algo. Its the Y! webmaster PR nightmare but multiplied multifold. Once people pay, Google may feel compelled to favour these pages or keep them in if the next day they intrdcue a filter that filters them out.
As others have pointed out, PFI does take money to run.
On the other hand other options do exist in the form of Adwords and Froogle. In fact there you get free guaranteed instantaneous (at least in the case of adwords) crawling!
Google's main unique advantage is the quality of their index (despite whatever roadwords are under way now) increasingly combined with a fast loading search page. Google strategically does not need to make money off the main SERPS. That is their come-on. They make money off adwords, licensing, and later maybe news.google, froogle, and a heap of ad ons. Moral for me is - keep the main index as useful and simple for the user as possible - then sell the eyeballs on adwords, licensing, news, and the like.
I know ATW and Teoma and sometimes AV have good results, but I use them much less because i know the google search page comes up almost instantaneously, and the results are simply displayed and that the others take longer to load, and the results sometimes beg for post processing in the form of "themes" and the extra post-processing options they offer. Thats great for advanced users, but for simplistic people for me, my perceptions is still that good results are faster in Google.
Additionally as far as AV and ATW are concerned I suspect they will be used by OV to feature their paid listings, as they are now, but how they are used when fully integrated is anybody's guess. My guess is that the OV content will be further featured and the default search page for ATW will become busier and slower to load, even if each continues as their own sub brand. - the future is even less well known for these brands as it is for google.
(sorry for drifting off topic)
Its also interesting to note that AV and ATW both utilize PFI (as does Ink) though we know that at least Ink has improved their index multifold since they restarted crawling the web wider for non-paid for content. In all cases, people here are reporting good SERPS in all in recent months, though that hasnt carried over to referrals (users have not been attracted yet). This to me will be a good test of how PFI influences the quality of search for users in general. It will be fascinating to see if OV increases the influence of PFI listings, keeps them the same, or decreases it.
