Mapping an entire website

searching for all links

[StereoBuddha]

Hi - newbie here.

I am searching a site and I want to be able to see every page connected to a homepage.

Is there any way of using Google to uncover the complete map of a site?

For example, say I am interested in the site www.example.com, is there any search prefix I can put into google that will reveal a list of all pages connected to the homepage.

Eg

www.example.com/about
www.example.com/today
www.exmaple.com/pages/links

[edited by: trillianjedi at 12:39 pm (utc) on Nov. 2, 2005]
[edit reason] Examplified - please see TOS ;-) [/edit]

[trillianjedi]

Hi StereoBuddha and welcome to WebmasterWorld!

By "connected" do you mean linked to, or linked from?

TJ

[StereoBuddha]

Hi TJ :-)

It's tricky trying to explain...

Basically I want to be able to see the complete site map of a site, but I only know the home page address.

I want to know every page that is online that begins with the address (for example) www.example.com/

Any page that follows the backslash that isn't necessarily obvious when viewing the home page.

There's probably an obvious way of describing what I'm trying to do, but I don't know it!

[edited by: trillianjedi at 1:17 pm (utc) on Nov. 2, 2005]
[edit reason] Examplifying - see above [/edit]

[trillianjedi]

I want to know every page that is online that begins with the address (for example) www.example.com/

I'm not sure what you mean now by "online"?

If you want to see all of your pages that are in a search engines index, you commonly would use:-

allinurl:www.example.com

But that's not to say that SE has crawled all of your pages.

For a complete sitemap, if you have shell access to your webserver, you can run an open-source application to do this for you:-

[google.com...]

Please use "example.com" for any examples.

TJ

[StereoBuddha]

Apologies for the example.com thing - I realise spamlinking may be an issue. It really was an example only though - there's no SB website as far as I know...

Anyway, one last go at trying to articulate this!

Imagine I want to discover every page that exists beginning www.example.com/

I want to create a map of a site (not mine, someone else's) showing where all the pages sit in relation to the homepage.

Is there a prefix in Google or other engines I can use?

I thought i might be able to to something like "sitemap: www.example.com"

Thanks.

[trillianjedi]

OK got it.

No, you'll need to use a script or crawler application to do that.

There's one or two in here that will probably do the trick, or you could use the one I mention above:-

[dmoz.org...]

TJ