Forum Moderators: open

Message Too Old, No Replies

How Many Pages Are On The Web?

A Research Project May Help Us Find Out

         

engine

3:46 pm on Sep 13, 2011 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



How Many Pages Are On The Web? [edition.cnn.com]
A group called the World Wide Web Foundation -- appropriately founded by Tim Berners-Lee, who pretty much created the internet -- is on a quest to figure out, with some degree certainty, how big the internet really is.

With a $1 million grant from Google, the foundation plans to release the results of its online forensic search, called the World Wide Web Index, early next year, the foundation's CEO, Steve Bratt, said in a recent interview.

"The Web Index will be the world's first multi-dimensional measure of the Web and its impact on people and nations. It will cover a large number of developed and developing countries, allowing for comparisons of trends over time and benchmarking performance across countries."

Bratt stressed that it won't answer every question people have about the internet, but he hopes the index, which will be presented as a series of annual reports, will go a long way toward filling in some of the gaps.

topr8

4:43 pm on Sep 13, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



hopefully they'll obey robots.txt or else a great many sites will be blocking them anyway.

brotherhood of LAN

4:58 pm on Sep 13, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I'll save them a lot of time and effort. Take one case of a bad .htaccess redirect/rewrite, or bad canonicalization and you have infinite pages ;o)

Realisically though, not infite but limited ot the number of domains that happens on * the number of character length/permutations allowed for an HTTP GET request.

Or how about just <?php echo $_GET['somenumber'];?>.

They should approach Google for some philosophy on what they consider constitutes a unique page of information, although some of the Panda belligerents may prefer another source...