Forum Moderators: open
The generation of pages via a programming language (or Miva) isn't in itself a problem, and as far as I know there's no reduced crawling or weighing based on .cgi or .mv URLs.
Because '?' URLs are sometimes used for session ids they can cause duplicate problems in search engines. These are often called 'dynamic URLs'. Google can and does index these URLs, but the crawl depth (and possibly rankings) are reduced.
Your options are to get more PageRank and hope that Google indexes those URLs; switch your cart software to use robot-friendly URLs (without session Ids until the user needs to be logged in); generate a flat file version of the site at each update (some cart software does this); or just add some static content as SmallTime suggests. The last option is probably the easiest.
There happens to be an add-on module available that turns Miva pages into static, and it's not expensive at all. But that's not available to me, so I've just excluded the whole shopping cart with robots.txt because there's no way I can have any control over those dynamic pages.
The homepage is PR5 and the few static pages in place are in /directories/ which are also PR5, as are some of the pages in the root. No problem there, it works fine, it just limits and gives control over the number of pages linked to, which is good.
The other related flat pages that are being created will be going, sub-themed after a fashion, within those /directory/ areas - let's say it's something like ladies' widgets, men's widgets and children's widgets. The convenience of automatic generation of pages is lost, so it's a bit cumbersome, but without ability to control any dynamic content it's the only choice. And it's much more controllable in terms of internal linking, which should eventually have an effect on the PR of all the internal pages as the site grows. So loss of convenience is giving much more PR control - very much worth it.
It actually would lend well to a canonical structure of the simplest kind:
mens-widgets.example.com/casual-widgets/winter.htm
ladies-widgets.example.com/casual-widgets/summer.htm
But I have no idea what we'd do with the shopping cart, not having dealt with that before, and having hosting limitations. Meantime, the directory structure is being developed as the only alternative that was available with two different people working on it with different software, and it's turning out very well.
I would definitely check out the MIVA modules that are out there and see if they can be properly adapted - there are dozens of them, MIVA is very flexible with the add-ons.
And yes, Google will index the file extension without the question mark, one got in by accident when the site was moved to another server, before robots.txt was uploaded. It presented no problem with Google at all.
