Welcome to WebmasterWorld Guest from 54.146.217.179

Message Too Old, No Replies

site:example.com urls - getting a full list

     
10:09 pm on Sep 19, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 8, 2004
posts: 562
votes: 1


We're in the process of going from static to dynamic. We're going to 301 the static pages. I have a list of static pages, but the quantity does not match what google has when I do a site:example.com.

I've searched G's Webmaster Galleria and 'Got Nothin' for helping on this issue.

Does anyone have an idea on how I can get a list of these site:mydomain.com URL's from Google?

[edited by: tedster at 1:25 am (utc) on Sep. 20, 2007]
[edit reason] switch to example.com - it can never be owned [/edit]

3:57 am on Sept 20, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


If you've got a relatively limited number of directories, you can use site:example.com/directory-a/, site:example.com/directory-b/ and so on. This kind of granular look will give you more of the indexed urls for your domain when the total tops 1,000.
11:15 am on Sept 20, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


You can have a dynamic site and continue to use the old "static URLs". Set up a rewrite (not a redirect) on the server so that it translates the requested static URL into the dynamic server filepath where the content resides, without exposing what that filepath is.

Changing the URLs of your pages is almost always a bad idea.

1:40 pm on Sept 20, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 8, 2004
posts: 562
votes: 1


site:example.com/directory-a/
I've done this and gotten the list of URL's but it has the accompanying verbage and then the URL's. I'm trying to get a strict list of URL's.

rewrite (not a redirect)
I was thinking that 301 would be the way to go but you're saying that a mod-rewrite is better? I guess I will still need to make a list of old URL's and have the location of the new URL. A vast majority of these "new" pages are going to be right off the root and not in the sub-folder structure like they currently are.
4:09 pm on Sept 20, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 16, 2004
posts:176
votes: 0


There are various tools online that will generate a sitemap for you. You then have to do a bit of searching and replacing to get rid of the extraneous information, which takes about five minutes.

You have to hunt around for a tool that will crawl our entire site, however. Some of them stop after a certain number of pages, and those tend to be the ones that rank highest in Google (go figure).

This is how I put together a link of 600 URLs for an .htaccess file so that I could 301 everything after an update to my site.

4:23 pm on Sept 20, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 8, 2004
posts:562
votes: 1


Would you care to disclose where/how I can obtain such a piece of software?
6:06 pm on Sept 20, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Give Xenu LinkSleuth a go. I use that a lot.

It is one of the very few tools that Moderators here allow to be disussed.

1:54 am on Sept 21, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 16, 2004
posts:176
votes: 0


I'm sorry, I thought it was obvious you could find a tool for generating sitemaps by searching on Google.
4:15 pm on Sept 26, 2007 (gmt 0)

New User

5+ Year Member

joined:Feb 2, 2007
posts: 4
votes: 0


I too am looking for this information. My situation is a bit different.

I have the list from Xenu and I have my site map list, but I am trying to figure out the differences in what Xenu and my sitemap say and what Google shows as indexed. Google has about 20% more URLS than Xenu shows or I send to them in my sitemap.

I suspect a problem.

Michael