Page is a not externally linkable
amanS - 12:04 am on Feb 28, 2007 (gmt 0)
Think of the amount of (previously invisible) websites that have been added in the past 2 years with Google and Yahoo trying to out do each other. Each trying to claim they have more content. My site and many others have content that is our intellectual property. I have to get some legal agreement from the user to not steal this content. This puts me behind a login “firewall” and keeps bots out…
The invisible web is not full of listings which waste the users’ time. Google, Yahoo and others are constantly trying to figure out how to index this content because it is useful to the users, not to mention adding to their ad revenue.