spidering my own sites and collating the data

..or something like that

[4eyes]

I have about 100 customer sites to manage would like to produce a list of the URLs for each page on each site.

In my dreams, this would be done by a nice piece of software into which I feed all the top level domains. It then goes off and spiders the hell out my list, producing one big list of the URLs covering every site and every page within.

Is this just a dream, or is there something out there to so this.

BTW I don't have all these site on my server, they are scattered to the wind!

[rcjordan]

Check out this thread [webmasterworld.com], Zeus came to mind as a possibility when I read your post. Give us an update if you find anything promising (I might want to do it myself).

[Woz]

Actually, in this case ZEUS would not be a good idea as it only returns the root domain of any site it "accepts". Sorry RC..

Perhaps Xenu [home.snafu.de] might be a better choice. This is a link checker that ultimately gives you a list of URL in the site and a site map. You would have to do each site seperately but you could then save each report as an html page and build your own mini directory.

Also you can save each site's file for updating at a later stage.

And Free as well...

Onya
Woz

[4eyes]

thanks Woz

Xenu is just the job

I can scan each site, get a list of the URLs of each page, and then decide which to autosubmit, which to do by hand, which to leave etc

..and its free - wonderful stuff!

[Woz]

You're Welcome.