Search Engines

What are they using?

[fashezee]

I would like to know what hardware and operating systems the major search engines
are using?

I remember see a thread about this awhile back; it went soemthing like; X% of search engines
used X% of this s/w?

[GeorgeGG]

HEAD groups.google.com HTTP/1.0

HTTP/1.0 200 OK
Content-Length: 2689
Connection: Keep-Alive
Server: GWS/2.0
Date: Thu, 06 Mar 2003 02:29:49 GMT
Content-Type: text/html
Cache-control: private
Set-Cookie: PREF=ID=4549911252bcb073:TM=1046917789:LM=1046917789:S=7jjoa3T1JP349fr1; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com

HEAD altavista.com HTTP/1.0

HTTP/1.0 200 OK
Set-Cookie: AV_POS=pos=1046918014931; path=/; domain=.altavista.com;
Set-Cookie: AV_USERKEY=AVS03b82a986390380a1c21510000052; expires=Tuesday, 31-Dec-2013 12:00:00 GMT; path=/; domain=altavista.com;
Server: AV/1.0.1
MIME-Version: 1.0
Cache-Control: no-cache,no-store,max-age=0
pragma: no-cache
Expires: Thu, 06 Mar 2003 02:33:34 GMT
Set-Cookie: AV_MKT=1; Domain=altavista.com; Path=/; Expires=Thu, 01-Dec-1994 16:00:00 GMT
Content-Type: text/html; charset=ISO-8859-1
Content-Length: 9982
Date: Thu, 06 Mar 2003 02:33:34 GMT

HEAD search.msn.com HTTP/1.0

HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Date: Thu, 06 Mar 2003 02:34:37 GMT
P3P:CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 249

Don't know about the % part.

GeorgeGG

[fashezee]

What O/S is GWS and A/V?

How did you find that out?

[GeorgeGG]

Can I sticky you?

GeorgeGG

[jeremy goodrich]

GWS = Google Web Server - they wrote their own. I remember reading that, probably here

Either Altavista did that as well - or they got slick and configed theirs to respond as something proprietary.

Oh, btw, check out the server header checker at Search Engine World [searchengineworld.com] you'll find a link someplace near the bottom of the page...lots of cool stuff there.

[GeorgeGG]

jeremy_goodrich

Search Engine World

Do you know if they accept URL's for other tools or
who I might e-mail and ask/submit?

thanks
GeorgeGG

[jeremy goodrich]

That site is owned by Brett Tabke, same chap that owns / runs this one. :)

[GeorgeGG]

Ok, thanks

GeorgeGG

[martin]

[uptime.netcraft.com...]

It certainly is more reliable than header checking.

[Fischerlaender]

Google is running on several thousand standard Intel-based PCs with Linux as their operating system.

I know that a small and now completely irrelevant german engine is running on a single Intel-based PC with Windows 2000 Server and MS SQL Server 2000 as the database management software.

And I know from a computer science professor, who is developing an engine for Germany, that they are using a cluster of 16 standard PCs to crawl, index and serve about 100 million pages. They are using SuSE Linux with some small patches of their own. Of course all the database code they use is written from scratch in C/C++.

IMHO most of the newer engines tend to use PC-Cluster because of the small price and the inherent stability of a cluster-based solution.

BTW: The Server-Section of the Headers can easily be modified by the engines. So they could use plain Apache but tell us that they are using their own ...