Welcome to WebmasterWorld Guest from 34.204.173.36

Forum Moderators: phranque

Message Too Old, No Replies

spidering my own sites and collating the data

..or something like that

     
6:07 pm on Apr 25, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 8, 2001
posts:1402
votes: 0


I have about 100 customer sites to manage would like to produce a list of the URLs for each page on each site.

In my dreams, this would be done by a nice piece of software into which I feed all the top level domains. It then goes off and spiders the hell out my list, producing one big list of the URLs covering every site and every page within.

Is this just a dream, or is there something out there to so this.

BTW I don't have all these site on my server, they are scattered to the wind!

6:28 pm on Apr 25, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member rcjordan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 22, 2000
posts:9138
votes: 0


Check out this thread [webmasterworld.com], Zeus came to mind as a possibility when I read your post. Give us an update if you find anything promising (I might want to do it myself).

Woz

11:46 pm on Apr 25, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 13, 2000
posts:4823
votes: 0


Actually, in this case ZEUS would not be a good idea as it only returns the root domain of any site it "accepts". Sorry RC..

Perhaps Xenu [home.snafu.de] might be a better choice. This is a link checker that ultimately gives you a list of URL in the site and a site map. You would have to do each site seperately but you could then save each report as an html page and build your own mini directory.

Also you can save each site's file for updating at a later stage.

And Free as well...

Onya
Woz

1:05 pm on Apr 26, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 8, 2001
posts:1402
votes: 0


thanks Woz

Xenu is just the job

I can scan each site, get a list of the URLs of each page, and then decide which to autosubmit, which to do by hand, which to leave etc

..and its free - wonderful stuff!

Woz

1:13 pm on Apr 26, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 13, 2000
posts:4823
votes: 0


You're Welcome.