Dynamic Site

Not getting spidered with Dynamic ASP Pages

[Ledfish]

I have an e-commerce site. Most pages are dynamic created with formats like www.webmasterworld.com?category=2 or www.webmasterworld.com?product=396. This site has been in place for about 4 months and google has not indexed any of these pages. Originally the category menu was dynamically created. To help I recently changed that to be hard programmed, but haven't seen any results yet. Naturally all the non-dynamic content has been indexed. Any thoughts?

[Chicago]

Welcome to WebmasterWorld, Ledfish.

dynamic asp pages can be a hit or miss proposition. we started to rewrite our dynamic asp ecommerce pages recently with unbelievable results on previously unindexed pages.

try the ISAPI_Rewrite - and focus on keyword rich url extentions with sound internal linking structure.

good luck/

[EricGen]

I have exactly the same problem with a php site. Any ideas about that problem, or do we have to consider creating a page per product?

[Ledfish]

Create a static page per product, that would really be a tough gig when you've got of 2500 of them in about thirty different categories. Throw in multiple options on most products and I could be just building those static product pages for so long, I wouldn't have time to service the orders that I am getting.

You know, I hear that google indexes dynamic pages, but I don't believe it. The whole problem makes people resort to doing things that are not just on the level.

[too much information]

I have 3 dymanic sites and so far no problems with any of them being crawled.

What you may want to look out for is the name of your querystring variables. Don't use 'ID' or a similar variable that might be mistaken for a session tag.

Just a week or so ago, Google crawled a dynamic directory that I created, it's really one page and a database feed depending on the querystring. I had a link to each variation as a 'site-map' and the bot hit every page.

[skipfactor]

>>You know, I hear that google indexes dynamic pages, but I don't believe it.

I believe it. Google's gobbled thousands of my dynamic pages on many different sites ranging in PR.

Ledfish, be sure the bot can open the file that connects to your database. Also stick a static link on your homepage to one of your dynamic pages & check your logs to see if Googlebot follows & indexes it. Also add the meta:

<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">

added: I haven't tested it, but I would guess that banning your connection file/folder in your robots.txt or password-protecting it might keep the bots from initiating it.

[edited by: skipfactor at 6:28 pm (utc) on Nov. 19, 2003]

[Ledfish]

Too Much

id in the string might be part of the problem and almost impossible to change because the shopping cart is third party.

[skipfactor]

I don't think it's "id=". GoogleGuy says to use something other than "id=" but I have and "id=" site that gets crawled like nobody's business.

[Ledfish]

Mine actual uses ones like idProduct= or idCategory=

[skipfactor]

Might not help you with a 3rd party cart, but I noticed a few interesting things with regard to Googlebot and what comes after the id:

[webmasterworld.com...]

[Ledfish]

Skipfactor

Thanks for that thread, it makes me wonder if the crawl of the dynamic pages isn't happening because of my page rank, which is probably not all that great,...yet. It's a theory at least and that's better than a shoulder shrug!

Luckily, my id='s are now more than 4 digits and start at 1.

I also want to say thanks to everyone who is participating in this thread, your thoughts and ideas are very help and hopefully this thread will be of help to others.

[skipfactor]

>>crawl of the dynamic pages isn't happening because of my page rank

That could be the case if you have a lot of pages indexed for the site; the bot hasn't gotten around to the dynamic ones yet though 4 months seems a bit long.

I would:
-Put a static link to a dynamic product page(preferrably the id=1) on highest PR page & watch the logs.
-Build PR, rinse, repeat
-Do my own shopping cart

[Birdman]

I second Chicago's suggestion. I don't see why it wouldn't work with third-party carts.

EricGen, you can do the same thing with Apache mod_rewrite.

I have rewritten URLs for a few sites that could not get crawled and it works every time.

[EricGen]

I have the situation described by skipfactor: the files with the db connection details is in a subdirectory disallowed in robots.txt, so I changed that to see if it makes a difference.

I don't know anything about mod_rewrite (I'm new to this) any pointer to explanations about what it is and how to use it?

Thanks

[skipfactor]

>>>added: I haven't tested it, but I would guess that banning your connection file/folder in your robots.txt or password-protecting it might keep the bots from initiating it.

>>I have the situation described by skipfactor: the files with the db connection details is in a subdirectory disallowed in robots.txt, so I changed that to see if it makes a difference.

Keep an eye on your logs & please let us know how this works out. I've been curious about it but too afraid to try it on a live site. ;)

[Ledfish]

I'm not using a robots.txt file, at least not yet, so I don't figure this has anything to do with my problem

[EricGen]

It seems to have done the trick for me. A search for a product this morning only gave the page where it is bundled (a static page). This afternoon it gives me the product's dynamic page. It's great thanks. I'm amazed it was that quick actually, the site's new and still has only PR 0 or none