The Deep Web

How big is it?

[kittykat]

Okay, we have heard about the Deep Web, the great unindexed mass of pages that search engines never get to. password protected pages, databases, and other content that, for whatever reason, is not indexed.

I saw a figure from a couple of years back, in 2001, that said the Deep Web was 500 billion pages big. How big do yall think it is now?

Of course, a lot has changed...Google and other search engines are better able to index dynamic content, etc. Nevertheless, the fact is that Google and others only touch the surface of the web -- probably less than 1 percent.

Any guesses on the percentage of the web covered by SEs?

Any ohter comments on the Deep Web?

[ritualcoffee]

well - since 2001 engines have indexed images, pdfs, etc. I think the largest part of the invisible web is now dynamic content. However, more and more dynamic sites are creating SE friendly pages. Also, PPC advetising has increased over a 100% - so is the invisible stuff as big as we think - I don't think so, I think it is shrinking.

[takagi]

There are now more pages with restricted access than a few years ago. Archived news pages for subscribers only. And a lot of bigger companies now have an intranet. Depends on your definition of 'Deep Web'.

[ritualcoffee]

great point - large amount of sites now have subscription areas. also - extranets.

[martinibuster]

Don't forget the millions, maybe billions, of groovy flash based hair salon web sites that are lost because they can't get indexed.

[mtaco]

Well, in our case, Google was only indexing 343 of our pages out of 160,000 because our site was dynamic.

I rewrote our link strategy recently, and now its up to 8,200.

Altavista did all 160,000 (though they haven't shown up yet).