homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

site:example.com urls - getting a full list

 10:09 pm on Sep 19, 2007 (gmt 0)

We're in the process of going from static to dynamic. We're going to 301 the static pages. I have a list of static pages, but the quantity does not match what google has when I do a site:example.com.

I've searched G's Webmaster Galleria and 'Got Nothin' for helping on this issue.

Does anyone have an idea on how I can get a list of these site:mydomain.com URL's from Google?

[edited by: tedster at 1:25 am (utc) on Sep. 20, 2007]
[edit reason] switch to example.com - it can never be owned [/edit]



 3:57 am on Sep 20, 2007 (gmt 0)

If you've got a relatively limited number of directories, you can use site:example.com/directory-a/, site:example.com/directory-b/ and so on. This kind of granular look will give you more of the indexed urls for your domain when the total tops 1,000.


 11:15 am on Sep 20, 2007 (gmt 0)

You can have a dynamic site and continue to use the old "static URLs". Set up a rewrite (not a redirect) on the server so that it translates the requested static URL into the dynamic server filepath where the content resides, without exposing what that filepath is.

Changing the URLs of your pages is almost always a bad idea.


 1:40 pm on Sep 20, 2007 (gmt 0)

I've done this and gotten the list of URL's but it has the accompanying verbage and then the URL's. I'm trying to get a strict list of URL's.

rewrite (not a redirect)
I was thinking that 301 would be the way to go but you're saying that a mod-rewrite is better? I guess I will still need to make a list of old URL's and have the location of the new URL. A vast majority of these "new" pages are going to be right off the root and not in the sub-folder structure like they currently are.


 4:09 pm on Sep 20, 2007 (gmt 0)

There are various tools online that will generate a sitemap for you. You then have to do a bit of searching and replacing to get rid of the extraneous information, which takes about five minutes.

You have to hunt around for a tool that will crawl our entire site, however. Some of them stop after a certain number of pages, and those tend to be the ones that rank highest in Google (go figure).

This is how I put together a link of 600 URLs for an .htaccess file so that I could 301 everything after an update to my site.


 4:23 pm on Sep 20, 2007 (gmt 0)

Would you care to disclose where/how I can obtain such a piece of software?


 6:06 pm on Sep 20, 2007 (gmt 0)

Give Xenu LinkSleuth a go. I use that a lot.

It is one of the very few tools that Moderators here allow to be disussed.


 1:54 am on Sep 21, 2007 (gmt 0)

I'm sorry, I thought it was obvious you could find a tool for generating sitemaps by searching on Google.


 4:15 pm on Sep 26, 2007 (gmt 0)

I too am looking for this information. My situation is a bit different.

I have the list from Xenu and I have my site map list, but I am trying to figure out the differences in what Xenu and my sitemap say and what Google shows as indexed. Google has about 20% more URLS than Xenu shows or I send to them in my sitemap.

I suspect a problem.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved