Invisible Web for Commerical Use

What now for search engines?

[worklive]

The invisible web is a massive archive of mostly dynamically generated web pages from database queries. It has very detailed commercial data and is much much vaster than the 'surface web' we currently communicate through. Has anyone used the invisible web commercially?

[Brett_Tabke]

I think we have to talk specifics. The general term "invisible web" is too generic and all emcompassing. Sites that traditionally don't get indexed that I've used:

- census data.

There are some large tracks of state, local, and federal (US) census data that doesn't get indexed. Some of that data is deep and detailed. (great kept secret from the general web searchers). I've found it of value on many occasions while researching demographics for clients. I can't count the times I've heard, how did you find that out? when making a proposal about some obscure fact right in their own backyard.

Dig through some of the state government and state university sites to find some of the census data. Much of it, isn't indexed by search engines.

- university sites.

Althought Googles .edu search has been a boon to the research crowd, it is still a very shallow indexing of sites. Then there is the inheritent weakness of Google because it only indexs 101k of each page. On many research docs, that is only just the introduction.

The problem is finding the really good data buried down in those edu sites. You have to visit many sites, read a great deal, and often dig through layer after layer of department documentation. The only saving grace is that most edu departments still practice bare bones html.

- The whole hidden database thing.

This is where it pays to be a surf junky with a fast internet connection and a ultra fast browser.