| 10:34 am on Aug 20, 2003 (gmt 0)|
nice find as always.
u cant be that busy working then :)
| 10:52 am on Aug 20, 2003 (gmt 0)|
starting to see a bit of traffic from Fast these days, and started using them more less spammy well done to the Fast Team
| 11:03 am on Aug 20, 2003 (gmt 0)|
3bn pages, the crawler has been busy lately.
>see a bit of traffic from Fast these days
That's interesting Dave. Care to elaborate a bit on that? Better rankings, more pages indexed?
| 11:21 am on Aug 20, 2003 (gmt 0)|
>>less spammy well done to the Fast Team
I am sure there are a few WebmasterWorld members that can sort that out :)
| 12:23 pm on Aug 20, 2003 (gmt 0)|
I really like the SERP's on alltheweb at the moment.
Not just for my site rankings but for general searches they seem to be damn good.
| 6:16 pm on Aug 20, 2003 (gmt 0)|
I agree the serp's are high quality.
| 8:22 am on Aug 21, 2003 (gmt 0)|
I do not see much relevance to our world by FAST or WiseNut or Teoma. But FAST especially: Who really uses FAST (All the Web)? At least the others are used by some premier search engines like Ask.
I can't see any competition to Google, Yahoo, MSN and AOL, and Ask, in that order, by anyone else in the SE world so why this attention to FAST?
Plus Google provides re****s to Yahoo, and AOL; MSN is ........what? -- Inktomi and Looksmart and its own MSN editors? It gets searches because the MSN homepage is the default opening page on every computer in the world using Windows. But it is not a search engine worth the name, otherwise.
But anyway: who uses All the Web? Why would any webmaster take the time to optimize for FAST? Escept to hedge against a near impossible success story.
Am I missing something here?
| 8:24 am on Aug 21, 2003 (gmt 0)|
who uses All the Web?
Enough people to pay for the indexing of 3,048,540,854 pages, at the least. :)
| 8:29 am on Aug 21, 2003 (gmt 0)|
Webstart, the answer to your question is here:
That was the past: Fast feeding portals, with ATW being a mere showcase and engine of choice for information professionals.
Today Fast is out of the equation. Their websearch technology and outlet, ATW, belongs to Overture, and soon to Yahoo.
What will happen to ATW and the underlying worldclass search technology is totally unclear at this point.
The fact however that ATW has grown their index from ~2 Bill to ~3 Bill seems to suggest there is still a future for ATW.
| 5:50 pm on Aug 21, 2003 (gmt 0)|
Rumbas, no more pages, ranking pretty much the same first page anyways.
We are just seeing more traffic, I think more people starting to use ATW, we track referrer to checkout on all orders and we are seeing people ordering our goods on ATW, yahoo down slightly and google, well depends if home page is in the serps or not but we where #1 for 18 months and our brand carries more traffic from google than the keywords, so unless we get our brand kicked from G I'm not that bothered, also started to seeing brand searching in ATW maybe google users are switching either anyway Well Done ATW Team!
| 6:31 pm on Aug 21, 2003 (gmt 0)|
I also noticed a change in SERPs quality in ATW. I am actually starting to use it once in a while. I am sure there is future for ATW. They just need to advertise and get on to the average user.
| 4:48 am on Aug 22, 2003 (gmt 0)|
May be 3 billion, but it is an old 3 billion. This is the main problem I see with all the alternatives to Google - stale results. AV updates more often than Fast
| 4:50 am on Aug 22, 2003 (gmt 0)|
Currently searching 3,151,743,117 web pages
So Google is no longer number 1.
| 7:08 am on Aug 22, 2003 (gmt 0)|
|May be 3 billion, but it is an old 3 billion. |
According to our logs, Fast is fetching 3 times as many pages as Googlebot the past 45 days. At this rate it won't be Fast that has stale results in the near future. If things stay the same, it will be Google that has the stale results if their spidering patterns don't change.
To me, it's really not the stale results that are the biggest issue. It's the 404 errors that come from a database that are not kept up to date. If a crawler doesn't check all pages often enough, there are going to be large numbers of removed pages that will be coming up on the first page of results. At one time, that page was very important to the index, but as time goes by, pages are removed from the Internet, but many search engines just keep listing them as there and important. This is the very reason I stopped using AV years ago.
[edited by: MarkHutch at 7:25 am (utc) on Aug. 22, 2003]
| 7:13 am on Aug 22, 2003 (gmt 0)|
I agree ATW results are great in every term we test for...
However, size of the database is to me, only a positive until a certain extent. Im sure there are millions of pages out there that are not useful to anyone to index.
Several years back, I remember a media frenzy about the "best search engine" on the net, (Google was still a baby then). Northern Light was then announced the "biggest" search engine as defined by number of pages indexed, eclipsing AV, Excite, Infoseek by far. It was the first time many in the mainstream had heard of Northern Light, and they got a lot of traffic for a limited time as a result. Look where they are now? In short "index size" is easy for the mainstream searcher to undestand and its easy to write a good headline about for the hacks out there.
Im not arguing against how good ATW is at all. I see their results as on a par and in many cases, superior than Googles at this time. I just question how important "index size" is for any search engine when it could be argued that most pages on the web are crud anyway... from duplicate content to doorway pages to p*rn to just plain poorly written, low credibility material.
To me its how useful the SERPS look to me when i try various queries where i need to get info. On that creitria ATW is very very good, but i doubt if its related to "index" size to any significant extent.
Now "freshness". Thats another important thing too, and ATW is performing well there too.
| 8:49 am on Aug 22, 2003 (gmt 0)|
Press Release here:
Just love how they don't give credit to WebmasterWorld!, and no offense to Greg, but it isn't even mentioned on his site, why don't just use BT as a reference, as it was probably said here first! and i wonder if the PR people at OV had any idea what so ever that they Alltheweb had reached the mark.
| 8:50 am on Aug 22, 2003 (gmt 0)|
Good points, Chiyo.
I think there are some points to be made for index size though, with the most important being promotional effects. It's a great PR tool being able to claim the largest index. We have seen this used in favour of Google over and over, it's something you can feed to all mainstream journalists. Everybody understands big numbers.
Technically having a large index is probably important for accuracy of ranking mechanisms, which take linking patterns into account.
Another point to consider is the permanent growth of the web. Having a 1 Bill docs index in spring 2002 means probably you have the same slice of the web indexed as with a 3 Bill. index in atumn 2003.
This has also to do with the explosive growth of large parts of the web in countries where webpublishing is just starting to become a mass phenomenon.
Also Google as well as ATW now have much more filetypes to index.
If in say two years we still have general search engines with the quest to index all_the_web I expect them to hold indexes of well over 10 Bill, perhpas much more.
| 10:00 am on Aug 22, 2003 (gmt 0)|
The point here is surely that alltheweb have some aggresive plans and funding. Once they can collect the info they can play with their algo.
| 4:54 pm on Aug 23, 2003 (gmt 0)|
Index size, Freshness, Relevancy ... anyone remember openfind [webmasterworld.com]?
Size doesn't allways matter. ;)
| 4:54 pm on Aug 24, 2003 (gmt 0)|
|According to our logs, Fast is fetching 3 times as many pages as Googlebot the past 45 days. At this rate it won't be Fast that has stale results in the near future. If things stay the same, it will be Google that has the stale results if their spidering patterns don't change. |
Fetching pages is different that updating the SERPS. Just took another look. Fast has succeeded in listing links made to my site in the past two weeks, but they have not added some 100+ pages of content added to my site over the last four months.
Teoma, AltaVista and Google all have done a better job of adding these new pages to their index.
I'll stand by my original statement - "May be 3 billion, but it is an old 3 billion."
| 5:59 pm on Aug 24, 2003 (gmt 0)|
I'm not sure I believe the 3.1 billion figure from ATW... For my site, it says 1 - 100 of 3,979 Results. But I only have around 2000 pages.
| 2:42 am on Aug 25, 2003 (gmt 0)|
>>Why All the Web?
I find it does catch my site changes quicker than Google; I also find it's results more consistent, but at the same time - closer to Google's results than any of the other search databases.
That's just my perception, and for the record I don't watch ATW serps as closely as I do Google's.
BTW, I optimize for Google and do great in both.
| 9:09 am on Aug 25, 2003 (gmt 0)|
That's great...3.2 billion pages being served to me and the other 2 people that use this engine.
It's a shame, because it gives quality results. It's a shame Yahoo didn't end up buying THIS instead of Ink.
| 9:10 am on Aug 25, 2003 (gmt 0)|
>> It's a shame Yahoo didn't end up buying THIS instead of Ink.
Looks like they will though, Yahoo is buying Overture which includes Alltheweb
| 8:07 pm on Aug 25, 2003 (gmt 0)|
Good point, and one that slipped my mind....
What are the odds Yahoo will Frankenstein some kind of Ink/AlltheWeb results?
| 8:25 pm on Aug 25, 2003 (gmt 0)|
No idea, really. Looks like Y! is set to use Ink - no idea why. ATW is bigger, better, faster, cheaper.
Another option could be to let ATW run as seperate unit, make it the lean and decidedly unportalish alternative. Why not cover all facettes of the search market?
| 9:20 pm on Aug 25, 2003 (gmt 0)|
Is there a free way to make alltheweb.com refresh or return to your pages to reindex them? I am already listed there quite a bit but I have added a lot more pages of content I would like indexed.
| 2:39 am on Aug 26, 2003 (gmt 0)|
|Now official: Currently searching 3,151,743,117 web pages |
So Google is no longer number 1.
Google has updated their homepage:
|©2003 Google - Searching 3,307,998,701 web pages |
| 6:14 am on Aug 26, 2003 (gmt 0)|
After G's counter has been on the same number for months, it's nice to know they had some many extra pages indexed they weren't telling us about.
...and let us hope those other 300 million pages have some real content, and are not all from one of my competitor's new websites
| This 49 message thread spans 2 pages: 49 (  2 ) > > |