I have a site that is going live very soon and cannot wait. It has a several parked pages that are doing well on the search engines.
Basically upon inspecting, all the pages are hidden in folders. Eg. cart/abc.php or html/abc.php This absolutely makes no sense to the search engines and as you can see these are from a programmer's point of view. I would love to mod rewrite it but I would be a major task with the number of pages.
I am thinking of just robot.txt no follow the complete site and switch over live from the old main page to the new main page only (which would the gateway to access all the other pages that have been no follow indexed temporarily). That mean the only new content would be the new main page.
I really don't want all the mucked up page addresses to be indexed only to move them later.
Do you think there is any problem with this in the eyes of the search engines? I don't know enough to know if I am breaking any rules or if it is not ethical etc.
all the pages are hidden in folders. Eg. cart/abc.php or html/abc.php
What do you mean by "hidden in folders"? There are numerous sites on the web with crazy URLs. I can see a problem if the content is hidden in a database and can only be retrieved after running a search *and* there is no browsable list of the products. I can also see a problem if the URLs have lots of slashes, but then again, I know well-indexed sites with as many as 14 slashes in the URLs. :( Google is good with long URLs now, but for a new site it is worth keeping them short and sweet.
Your mention of mod_rewrite suggests an Apache server. Have you visited our Apache forum [webmasterworld.com]? One of the prior discussions there might help.