Google needs to improve it searching of Dynamic Content

Yahoo Can Do it, MSN Can Do it, Why is Google so bad at it

[lgn1]

Our Dynamic Site is in the form

www.widget.com/cgi/cgi-script.pl?parameter1=variable1&parameter2=variable2

Google is unable to spider my dynamic site on this.

In the past year, both MSN and Yahoo, will deep spider my site based on this dynamic structure.

We have been at it for five years, and we are a leader for our niche market, so Google would be spidering our site if it could, so PR and popularity is not an issue.

Any chance that Google will put priority on spidering dynamic sites, now that its competition is doing it with ease.

[wanderingmind]

I had a dynamic site with the same URL structure, but Google used to spider it quite well.

However, in the last 2 months, that seem to have stopped, and all I have now in SERPs are URLs of 50,000 dynamic pages.

Nothing has changed on the site, so either Google has changed its rules somewhere, or Googlebot has lost a bit of its power to spider dynamic sites deeply.

[MHes]

Dynamic content can be high quality but all too often it is rubbish. Thus google has to be wary and perhaps will only go deep if the pr is high enough.

The other factor is dynamic content is sometimes very focused... perhaps too focused. e.g a page may show the price and pic of one make of widget. For a search on 'widget' logic dictates that a more general page better serves the user than listing specialist ones. Hence a home page is a better page to serve than a deep one, thus giving the user a path to more options.

[jaffstar]

I am noticing that many .ASP pages are not getting crawled. Is this common?

[flobaby]

Just this weekend I learned about mod_rewrite which got rid of all those long dynamic phrases and immediately Google (and MSN) started gobbling them up! The next day I had 6 times the pages indexed that I used to.

I highly recommend it.

[edited by: flobaby at 6:17 pm (utc) on Nov. 10, 2004]

[WebFusion]

It's so easy to convert dynamic URls into static appearing ones that it's rediculous to complaing about it and not just have a programmer take the couple of hours to build that functioanlity into your system.

We have over 500 dynamic product URls on our site, yet all the search engines (and visitors, for that matter) see them as:

www.widgets.com/store/prod21.html

Waiting for the engines to become dynamic URL friendly is leaving money on the table. Think where your site would be now if you had done the necessary reprogramming this time last year! Don;t make the same mistake for 2005.

[JuniorOptimizer]

Making the URIs spider friendly is a trivial task. If you're not tech-savvy, you can outsources the job for $20 or less.

[lgn1]

It is trivial under apache, but not under IIS, especially if your ISP will not install ISAPI_REWRITE
or the equivalent.

[WebFusion]

It is trivial under apache, but not under IIS, especially if your ISP will not install ISAPI_REWRITE
or the equivalent.

Then why not use this as an excuse to move to apache? Sounds like IIs is too limited for what you NEED to do.