googlebot and cgi -- do the jive?

pages generated by a database

[askjoe]

I have a website that takes html, pictures, prices etc from a database using cgi. When someone clicks on the link "widgets" a cgi script runs to create the page. Can googlebot see this "generated on the fly" page even though there is no real "hard" copy? The text is in html and all the features such as "title" etc are in the database.

My question is will my site get indexed when all the pages within my site are generated in this fashion? When the googlebot reaches this junction, does it go into the database and spider the content? If not, I'm going to have to create them by hand which suck because I have a ton of products.

[mcavic]

Spiders will see the generated pages just like people do. They don't really know the difference between between static pages and dynamic.

But, if your site doesn't have a high enough PR, Googlebot might not follow all of your links. I'd wait until the end of the current deep crawl and see what happens. As long as Googlebot can follow your links, it should be fine.

[ruserious]

I think you have somewhat warped image of how spiders work. Basically with the web a page request is a page request, it doesn't matter wether it's a human with an IE clicking on a link, or wether it's an automated programm on some server requesting the page. So a spider does not see or know more than a "normal" human user.

As mcavic wrote the cgis execute server side, but the person/programm/bot who sees the output page does not know/care wether it's static or dynamically renerated.

[askjoe]

My biggest concern was the fact that the googlebot might not be able to see the content of the html within my database - if it can't see it, it can't spider/index it. Sounds to me like it can be seen.

But I have another question for you. How come when google posts the url for a listing, there's no "cgi" or "asp" within the url. Now, I know when I surf someone's site - I come across urls with these functions in them in the browser/command line all the time. This is what concerns me.

[mcavic]

It doesn't have to look into the database - all it has to do is follow the links and it'll get the content. That's the real beauty of server-side scripts like cgi and php.

How come when google posts the url for a listing, there's no "cgi" or "asp" within the url.

If you mean that when you search Google you don't usually see URLs with cgi or asp, that's true - it seems like most listings are static pages. There are dynamic listings, though - if you search for "view.php", you'll find some.