Forum Moderators: open
Says itself, well done FAST team.
I think this will probably make google change the number on their frontpage in not to distant future.
u cant be that busy working then :)
Shak
Dave
>see a bit of traffic from Fast these days
That's interesting Dave. Care to elaborate a bit on that? Better rankings, more pages indexed?
I am sure there are a few WebmasterWorld members that can sort that out :)
Not just for my site rankings but for general searches they seem to be damn good.
I can't see any competition to Google, Yahoo, MSN and AOL, and Ask, in that order, by anyone else in the SE world so why this attention to FAST?
Plus Google provides re****s to Yahoo, and AOL; MSN is ........what? -- Inktomi and Looksmart and its own MSN editors? It gets searches because the MSN homepage is the default opening page on every computer in the world using Windows. But it is not a search engine worth the name, otherwise.
But anyway: who uses All the Web? Why would any webmaster take the time to optimize for FAST? Escept to hedge against a near impossible success story.
Am I missing something here?
Enough people to pay for the indexing of 3,048,540,854 pages, at the least. :)
Jordan
That was the past: Fast feeding portals, with ATW being a mere showcase and engine of choice for information professionals.
Today Fast is out of the equation. Their websearch technology and outlet, ATW, belongs to Overture, and soon to Yahoo.
What will happen to ATW and the underlying worldclass search technology is totally unclear at this point.
The fact however that ATW has grown their index from ~2 Bill to ~3 Bill seems to suggest there is still a future for ATW.
We are just seeing more traffic, I think more people starting to use ATW, we track referrer to checkout on all orders and we are seeing people ordering our goods on ATW, yahoo down slightly and google, well depends if home page is in the serps or not but we where #1 for 18 months and our brand carries more traffic from google than the keywords, so unless we get our brand kicked from G I'm not that bothered, also started to seeing brand searching in ATW maybe google users are switching either anyway Well Done ATW Team!
DaveN
WBF
Currently searching 3,151,743,117 web pages
May be 3 billion, but it is an old 3 billion.
According to our logs, Fast is fetching 3 times as many pages as Googlebot the past 45 days. At this rate it won't be Fast that has stale results in the near future. If things stay the same, it will be Google that has the stale results if their spidering patterns don't change.
To me, it's really not the stale results that are the biggest issue. It's the 404 errors that come from a database that are not kept up to date. If a crawler doesn't check all pages often enough, there are going to be large numbers of removed pages that will be coming up on the first page of results. At one time, that page was very important to the index, but as time goes by, pages are removed from the Internet, but many search engines just keep listing them as there and important. This is the very reason I stopped using AV years ago.
[edited by: MarkHutch at 7:25 am (utc) on Aug. 22, 2003]
However, size of the database is to me, only a positive until a certain extent. Im sure there are millions of pages out there that are not useful to anyone to index.
Several years back, I remember a media frenzy about the "best search engine" on the net, (Google was still a baby then). Northern Light was then announced the "biggest" search engine as defined by number of pages indexed, eclipsing AV, Excite, Infoseek by far. It was the first time many in the mainstream had heard of Northern Light, and they got a lot of traffic for a limited time as a result. Look where they are now? In short "index size" is easy for the mainstream searcher to undestand and its easy to write a good headline about for the hacks out there.
Im not arguing against how good ATW is at all. I see their results as on a par and in many cases, superior than Googles at this time. I just question how important "index size" is for any search engine when it could be argued that most pages on the web are crud anyway... from duplicate content to doorway pages to p*rn to just plain poorly written, low credibility material.
To me its how useful the SERPS look to me when i try various queries where i need to get info. On that creitria ATW is very very good, but i doubt if its related to "index" size to any significant extent.
Now "freshness". Thats another important thing too, and ATW is performing well there too.
Just love how they don't give credit to WebmasterWorld!, and no offense to Greg, but it isn't even mentioned on his site, why don't just use BT as a reference, as it was probably said here first! and i wonder if the PR people at OV had any idea what so ever that they Alltheweb had reached the mark.
I think there are some points to be made for index size though, with the most important being promotional effects. It's a great PR tool being able to claim the largest index. We have seen this used in favour of Google over and over, it's something you can feed to all mainstream journalists. Everybody understands big numbers.
Technically having a large index is probably important for accuracy of ranking mechanisms, which take linking patterns into account.
Another point to consider is the permanent growth of the web. Having a 1 Bill docs index in spring 2002 means probably you have the same slice of the web indexed as with a 3 Bill. index in atumn 2003.
This has also to do with the explosive growth of large parts of the web in countries where webpublishing is just starting to become a mass phenomenon.
Also Google as well as ATW now have much more filetypes to index.
If in say two years we still have general search engines with the quest to index all_the_web I expect them to hold indexes of well over 10 Bill, perhpas much more.
Doug
Size doesn't allways matter. ;)
According to our logs, Fast is fetching 3 times as many pages as Googlebot the past 45 days. At this rate it won't be Fast that has stale results in the near future. If things stay the same, it will be Google that has the stale results if their spidering patterns don't change.
Fetching pages is different that updating the SERPS. Just took another look. Fast has succeeded in listing links made to my site in the past two weeks, but they have not added some 100+ pages of content added to my site over the last four months.
Teoma, AltaVista and Google all have done a better job of adding these new pages to their index.
I'll stand by my original statement - "May be 3 billion, but it is an old 3 billion."
WBF
I find it does catch my site changes quicker than Google; I also find it's results more consistent, but at the same time - closer to Google's results than any of the other search databases.
That's just my perception, and for the record I don't watch ATW serps as closely as I do Google's.
BTW, I optimize for Google and do great in both.
It's a shame, because it gives quality results. It's a shame Yahoo didn't end up buying THIS instead of Ink.
What are the odds Yahoo will Frankenstein some kind of Ink/AlltheWeb results?
Another option could be to let ATW run as seperate unit, make it the lean and decidedly unportalish alternative. Why not cover all facettes of the search market?
Now official: Currently searching 3,151,743,117 web pages
So Google is no longer number 1.
Google has updated their homepage:
©2003 Google - Searching 3,307,998,701 web pages
:)
...and let us hope those other 300 million pages have some real content, and are not all from one of my competitor's new websites
:-)
