This is really maddening. We are trying to transfer our website with great rankings to another platform and split the products into two separate sites. No duplicates, of course.
The new platform is data-base driven with pages written on the fly. The old one used the good-old static pages.
Mod_rewrite is used to re-write SE friendly URLs for the new site.
Yahoo! is indexing the SE-friendly pages fine.
A sitemap is registerd with Google that has the SE-friendly URLS, not the long cgi-bin urls. Google Sitemaps says these SE-friendly pages are "Unreachable URLS" due to network issues. That's not true.
Robots are restricted from cgi-bin in robots.txt.
GOOGLE FAILS TO FOLLOW THE MOD-REWRITE pages. It is stubborn and keeping our old site pages that are static and dropping them as we transfer the static pages on the old site to the dynamic one on the new site.
We have 2 other websites with the exact same architecture and a totally different product line. These are indexed fine by google.
Can anyone help determine why Google is being so stubborn? What can we do to overcome this? We've been working on this all year and was hoping once Big Daddy rolled out, it would sort out. It hasn't.