Netcraft Web Server Survey

What's this one?

[Rollo]

Is was wondering who is behind this one?

Netcraft Web Server Survey

Is it worthy or worthy of of being disallowed?

[volatilegx]

Netcraft is a company that keeps statistics on which web servers are being used to host web pages, and various other metrics.

See [news.netcraft.com...]

[Rollo]

Thanks!

What is your opinion about the Harvest spider/project? The name alone makes me a bit weary... sort of like the Total Information Awareness program. From what I could find out it seems to be an abbitious government project... is that correct?

[Lord Majestic]

I could find out it seems to be an abbitious government project... is that correct?

No -- they are a small private company made of a bunch of good people from the UK. They have been around for a very long time and their analysis (that they conduct on the basis of their automated surveys) is pretty much universally respected.

The web stats in question are derived from single request to a domain name -- this is as light on bandwidth usage as it gets! While you are unlikely to see any direct benefits from their survey in terms of traffic send to you, it is a good netizen's behavior to allow this sort of activity. Indirectly you will benefit from the Net being better understood, and thus attracting more investment.

[Rollo]

Interesting. The reason I was thinking it is a scary US Gov project is becuase I read that it was funded by the Defense Advanced Research Projects Agency [en.wikipedia.org]

Here's where I got the info [ibiblio.org].

Not the same thing?

[jdMorgan]

Then again, before the internet was called "the internet," there was DARPAnet...

They decide to open it up, and this was the result.

Jim

[Lord Majestic]

> Not the same thing?

I might have missed something, but I failed to find references to NetCraft in the links above? And yes DARPA funds a number of projects that are not necesserily of direct military significance, I doubt they fund NetCraft however since NetCraft was around for so long that and they have established themselves in their own niche so well that they have a number of huge corporate clients that pay subscriptions on a regular basis and this should be enough to sustain their small, but very good company.

While their activity will not benefit your site directly, they are doing important PR job for the Internet, ie for us all, and since they are behaving themselves and don't burden you with cost of their activity I would personally recommend to avoid banning them.

[Rollo]

I'm sorry, I though your were talking about Harvest... yes, I belive you're right about Netcraft. The quotes were about Harvest which I slipped in later in the thread. Perhaps Harvest deservs its own thread, I wonder what that project is all about.

Thanks!

[volatilegx]

Umm let's keep this thread on-topic... Netcraft.