I need a Spider Simulator for my site

Sim Spider

[Randyd]

Does anyone know if I can get a spider simulator like the Sim Spider on WebmasterWorld. I would like to offer my visitors something like that. So they could put their URL in the http field and have it output a list of pages found on their site or the site they want to spider.

Thanks

[jamie]

hi randyd,

i am sure you could use the command line tool wget via a system() call.

be aware that you might be leaving yourself open to abuse and could be hitting some of the spidered sites pretty hard?

[Randyd]

Thanks for the reply.

I'm not sure I know what you mean. Only members would be using a script like this and IP would be logged so abuse would be minimal.

Could you explain a little further of what you mean. Call to a function?

Please direct me to where I should go next or what I should do.

Thanks a bunch!