Welcome to WebmasterWorld Guest from 54.198.87.238

Message Too Old, No Replies

site:example.com urls - getting a full list

     
10:09 pm on Sep 19, 2007 (gmt 0)

10+ Year Member



We're in the process of going from static to dynamic. We're going to 301 the static pages. I have a list of static pages, but the quantity does not match what google has when I do a site:example.com.

I've searched G's Webmaster Galleria and 'Got Nothin' for helping on this issue.

Does anyone have an idea on how I can get a list of these site:mydomain.com URL's from Google?

[edited by: tedster at 1:25 am (utc) on Sep. 20, 2007]
[edit reason] switch to example.com - it can never be owned [/edit]

3:57 am on Sep 20, 2007 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



If you've got a relatively limited number of directories, you can use site:example.com/directory-a/, site:example.com/directory-b/ and so on. This kind of granular look will give you more of the indexed urls for your domain when the total tops 1,000.
11:15 am on Sep 20, 2007 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



You can have a dynamic site and continue to use the old "static URLs". Set up a rewrite (not a redirect) on the server so that it translates the requested static URL into the dynamic server filepath where the content resides, without exposing what that filepath is.

Changing the URLs of your pages is almost always a bad idea.

1:40 pm on Sep 20, 2007 (gmt 0)

10+ Year Member



site:example.com/directory-a/
I've done this and gotten the list of URL's but it has the accompanying verbage and then the URL's. I'm trying to get a strict list of URL's.

rewrite (not a redirect)
I was thinking that 301 would be the way to go but you're saying that a mod-rewrite is better? I guess I will still need to make a list of old URL's and have the location of the new URL. A vast majority of these "new" pages are going to be right off the root and not in the sub-folder structure like they currently are.
4:09 pm on Sep 20, 2007 (gmt 0)

10+ Year Member



There are various tools online that will generate a sitemap for you. You then have to do a bit of searching and replacing to get rid of the extraneous information, which takes about five minutes.

You have to hunt around for a tool that will crawl our entire site, however. Some of them stop after a certain number of pages, and those tend to be the ones that rank highest in Google (go figure).

This is how I put together a link of 600 URLs for an .htaccess file so that I could 301 everything after an update to my site.

4:23 pm on Sep 20, 2007 (gmt 0)

10+ Year Member



Would you care to disclose where/how I can obtain such a piece of software?
6:06 pm on Sep 20, 2007 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Give Xenu LinkSleuth a go. I use that a lot.

It is one of the very few tools that Moderators here allow to be disussed.

1:54 am on Sep 21, 2007 (gmt 0)

10+ Year Member



I'm sorry, I thought it was obvious you could find a tool for generating sitemaps by searching on Google.
4:15 pm on Sep 26, 2007 (gmt 0)

5+ Year Member



I too am looking for this information. My situation is a bit different.

I have the list from Xenu and I have my site map list, but I am trying to figure out the differences in what Xenu and my sitemap say and what Google shows as indexed. Google has about 20% more URLS than Xenu shows or I send to them in my sitemap.

I suspect a problem.

Michael

 

Featured Threads

Hot Threads This Week

Hot Threads This Month