Dynamic pages indexed without "site map"?

paranoia about offering links to dynamic content

[redherring917]

For lack of a better quick and dirty explanation, we run a Web site that offers an industry-specific "yellow pages". The power of our Network lies in allowing the user to search our client listings by multiple layers of industry-specific keywords and / or geographical criteria, but the end result is always the same - the user will eventually arrive at a dynamically generated "company profile" page unique to one selected client.

We've initiated a paid inclusion XML feed campaign to have these thousands of profile pages indexed, and the results have been quite impressive...and expensive. We'd like nothing more than to get these pages indexed outside of a CPC scenario.

Revising the url's of these dynamic pages to appear to be static is not an issue, but frankly, providing a "site index" or navigational link index that points to each of these client listings scares me to death. I'm likely just paranoid, but I so strongly resist "opening the books" like this to our competitors.

So what's a guy to do? Getting pages, (dynamic or otherwise), crawled that are not linked to from anywhere sure sounds like Mission Impossible.

Thanks.

[takagi]

Hi redherring917, welcome to WebmasterWorld.

I don't understand one point in your explanation. What can a competitor learn about your site if there is a site index, that he cannot learn from the Google? If a competitor searches for

site:www.yourdomain.com -blablabla

he will see all the pages in Google's index that don't contain the word blablabla. If he searches for

site:www.yourdomain.com keyword1

the pages containing keyword1 show up. So a site index will not give that much extra information. And besides, the site index is not visible in the backward links if this index has a PR that is below PR4.

[redherring917]

Thanks for the welcome and the reply, takagi ("long time listener, first time caller").

The way our site is currently configured, a user / competitor could never ask for and receive a list of:

- all of our clients
- all of our clients in a particular state
- all of our cleints that offer a particular service
- etc. etc.

The structure of the site requires keyword and geographical search criteria that results in highly targetted and limited results to the user. This is

1. good for the user, and
2. helps protect us from data theft, (there's that paranoia again), and a competitor being able to easily determine that we have "x" clients.

The success of our XML feed indexing has been in users searching for specific company names and / or industry-specific keywords + geographical parameters (city / state). If our profile pages were to be indexed via a regular crawl outside of a PPC scenario, I expect that they would be found in a similar manner. I think it would be difficult or at least very time consuming for a competitor to glean information we don't want them to have via Googling.

But if we employ a site index, do we not greatly facilitate this unwanted access to large amounts of information by providing handy links to the profile page of every client on our site?

Thanks again.

[takagi]

So basically you want the site index only to get all your dynamic pages indexed. But you're afraid it could give your competitor some information. In such a case, there is no need to group the links geographically or any other way. There is no need to give logic link text. So you could make pages with 100 links on each page and put the links in a random order with link text 1, 2, 3 ... 100. That wouldn't give any clue to your competitor except for the number of pages in your site. But as explained in an earlier posting, that is information any smart Google user can find out once the pages are successfully indexed by Google.

[redherring917]

Thanks again.

Is this kosher? Non-logical link text? Makes sense, but would it hold up under scrutiny from the search engines / is it a good idea?

Seems to me we'd also "have to" link to the upper level of these pages of links from our home page for them to be crawled, but I'm not sure it would be too cool for our users to find themselves navigating through hundreds of pages of non-logical link text. Sure, we could provide a "hidden" link from our home page, but again, this is rather frowned upon, is it not?

[takagi]

> I'm not sure it would be too cool for our users to find themselves navigating through hundreds of pages of non-logical link text.
If you link these index pages from some boring page that is hardly visited by users (like the copyright page or other legal text you need on your site but no one ever reads) then most users won't see it. Google will find your index pages when these are linked from a page already in the index. Since there are no keywords on it, users will not find them in the SERP.

> ... "hidden" link from our home page, but again, this is rather frowned upon, is it not?
A hidden link could cause some problems. What I suggest is not hidden. The user can see it (but will soon hit the back button).

> Is this kosher?
It is on the border I think. GG advices to only do things that are good for the user. But you want to hide some information because the 'user' could be a competitor. Maybe some other members on this forum could explain how they think about it.

[redherring917]

I appreciate all of takagi's input, but I too would love to get some additional feedback from other members of the forum.

Thanks in advance.