newbie: crawling/indexing dynamic URL's

Why do some dynamic URL's get indexed and others dont?

[aagha]

I'm curious to know why it seems that a site like Google indexes some of the URL's on my site, and not other.

It seems that it has no problem finding our site (because I submitted it), but once it gets there, it doesn't seem to crawl any of the (dynamic) links. Furthermore, my index.html is a redirecto to a dynamic page. Could this be something keeping Google from crawling?

What about other search engines?

Any help here would be appreciated.

Thx in Advance,
Zeb

[edited by: ciml at 1:07 pm (utc) on Jan. 15, 2003]
[edit reason] Sorry, no .sig URLs. [/edit]

[Sinner_G]

Hi aagha and welcome to WW

If your redirect is made with a script (e.g. <script>location.href = "dynamic.asp"</script>), then googlebot simply can't read it.

If you put a regular link into a noscript tag, the bot will be able to read that one and follow it.

[lazerzubb]

aagha, Welcome to WebmasterWorld [webmasterworld.com]
Per TOS we don't use signatures.

When it comes to Google indexing dynamic url's there is a lot of different reasons why they will and will not crawl a dynamic url.

Try to keep the dynamic url as clear as possible.

If you use a url, that has something like index.asp?ID=1000 there is a big chance that it will not get indexed only due to the "ID" part since this is often used for tracking/sessions.
Try using the "Site Search" (On the top of the page), and search for something like "Google indexing dynamic url's".

Also as Sinner_G, says if you use JavaScript to redirect users, Google will not be able to follow it.

[MHes]

Hi

I think lowish pr sites (<3) will not have as many dynamic links indexed. The greater the pr of the home page, the deeper googlebot will go into a dynamic site......I think.

[aagha]

Excellent set of responses. Thx so much.

Based on other postings I've read on this site, I've decided to do the following:

1. Use mod_rewrite to rewrite my site URL's so they appear static so that:

[mydomain.com...]

will become:

[mydomain.com...]

2. Add a site index to my home page which has links deep into my site.

From what I gather from the numerous other postings on this site about this very same subject, these two things can only help.

Rgs,
Zeb

[TylerDurden]

Use mod_rewrite to rewrite my site URL's so they appear static so that:
[mydomain.com...]
will become:
[mydomain.com...]

I might be wrong but I've heard recently from my friend that Google has problems indexing multi-dot URLs. E.g. Apache language version pages are not indexed by Google.

I would suggest setup mod_rewrite in a way it presents pages as a directories, something like:

[mydomain.com...]
or
[mydomain.com...]

Look at the way this forum's URLs done.

[aagha]

I don't understand what kind of sense that would make. Besides, I have only one or two dots in my domain:

[mydomain.com...]

The one right before the "display", and the other before the "html". I think if they're going to exclude that, then well, 'fogettaboutit!

What are some examples (real URL's) of Apache lang pages that you're talking bout?

I've done quite a bit of research on the Google issue, and this is the first time I've come across this one.

- Zeb