site:example.com urls - getting a full list

[Propools]

We're in the process of going from static to dynamic. We're going to 301 the static pages. I have a list of static pages, but the quantity does not match what google has when I do a site:example.com.

I've searched G's Webmaster Galleria and 'Got Nothin' for helping on this issue.

Does anyone have an idea on how I can get a list of these site:mydomain.com URL's from Google?

[edited by: tedster at 1:25 am (utc) on Sep. 20, 2007]
[edit reason] switch to example.com - it can never be owned [/edit]

[tedster]

If you've got a relatively limited number of directories, you can use site:example.com/directory-a/, site:example.com/directory-b/ and so on. This kind of granular look will give you more of the indexed urls for your domain when the total tops 1,000.

[g1smd]

You can have a dynamic site and continue to use the old "static URLs". Set up a rewrite (not a redirect) on the server so that it translates the requested static URL into the dynamic server filepath where the content resides, without exposing what that filepath is.

Changing the URLs of your pages is almost always a bad idea.

[Propools]

site:example.com/directory-a/

I've done this and gotten the list of URL's but it has the accompanying verbage and then the URL's. I'm trying to get a strict list of URL's.

rewrite (not a redirect)

I was thinking that 301 would be the way to go but you're saying that a mod-rewrite is better? I guess I will still need to make a list of old URL's and have the location of the new URL. A vast majority of these "new" pages are going to be right off the root and not in the sub-folder structure like they currently are.

[Haecceity]

There are various tools online that will generate a sitemap for you. You then have to do a bit of searching and replacing to get rid of the extraneous information, which takes about five minutes.

You have to hunt around for a tool that will crawl our entire site, however. Some of them stop after a certain number of pages, and those tend to be the ones that rank highest in Google (go figure).

This is how I put together a link of 600 URLs for an .htaccess file so that I could 301 everything after an update to my site.

[Propools]

Would you care to disclose where/how I can obtain such a piece of software?

[g1smd]

Give Xenu LinkSleuth a go. I use that a lot.

It is one of the very few tools that Moderators here allow to be disussed.

[Haecceity]

I'm sorry, I thought it was obvious you could find a tool for generating sitemaps by searching on Google.

[GEMaffair]

I too am looking for this information. My situation is a bit different.

I have the list from Xenu and I have my site map list, but I am trying to figure out the differences in what Xenu and my sitemap say and what Google shows as indexed. Google has about 20% more URLS than Xenu shows or I send to them in my sitemap.

I suspect a problem.

Michael