Despite Mod Rewrite Google is Indexing URLs as Dynamic

[LOLDavid]

Hi. We've done a mod rewrite on just about all our URLs. For a long time Google seemed to spider all our pages as the rewrite pages, but just recently it seems that Google has taken to spidering our site as dynamic pages. We have Google Free Site Search on our site so you can check it out and you'll see all dynamic results.

The only way you can really get dynamic pages on our site is by blocking cookies which displays URLs with added on PHP session ID:

http://www.example.com/productpage.php?cat=1&catid=&level=2&subcatid=129&id=15594&nav=B&PHPSESSID=764f891b9a8a3e01dfba6da28668bfe7

Some of the dynamic URLs in Google results have this session ID but some are just our normal dynamic links. I'm wondering how and why Google is spidering our site this way when we have done a mod rewrite and would like Google to index our rewritten pages and not dynamic ones.

We're making a site map to submit to Google so hopefully that will help. I'm just worried that the reason that not all our site is being indexed is due to Google's spidering our site as dynamic. Any ideas as to why this might be happening?

<Sorry, no personal URLs or specifics.
See Terms of Service [webmasterworld.com]>

[edited by: tedster at 5:05 am (utc) on April 22, 2006]

[tedster]

Welcome to the forums David. A couple ideas come to mind:

1. No search engine spider takes cookies. When the spider user agent doesn't accept the cookie, what information are you giving it?

2. Have you changed the links on your site to the rewritten form?

[JuniorOptimizer]

It might be safe practice to exclude the dynamic pages via robots.txt

[JoaoJose]

You should redirect all your dynamic urls to the new static ones. If you can't get a pattern between dynamic and static urls that allow you to do this, just return a 404 for all dynamic urls and you'll be ok. If you already have dynamic inbound links try to contact the linkers in order to change the link.

[BillyS]

I think the right order is:

1 - Make sure mod_rewrite is working EXACTLY the way you want it to work.

2 - Redirect anything that looked like an old URL to the new URLs. If everything was done correctly, then you can normally stop here. Now is the time to pause and make sure indexing is proceeding the way you want it to (that is, all the important spiders are doing what you want them to do.

3 - Block the old URLs via Robots.txt.

[legallyBlind]

I changed a site from dynamic calls to actual static html files to minimize the load of the server's database engine about 3 years ago. The website has over 180,000 pages and guess what? I suffered from dual content for about 2 years, but my rank is finally back. It took google about 2 years to clear most of the mess and what I had to do is rename or delete the files with dynamic call forcing google to read my new static files.

good luck...