Where to get 'website spider' bot for PHP?

[irock]

Hi, I have a somewhat weird request, but I have a legit reason. My new CMS is having performance problems, and we are trying to resolve the issue by implementing a bot or spider that grabs the HTML of the site every 30s.

I was wondering if you guys could kindly tell me where to get such bots. The program will be installed on another server with cronjob to run every 15s.

The 'website performance' check services out there are limited to 2 minute interval. I need something that hits the target site in question more frequently.

(Argh... EZ Publish... such a pain.)

Thanks much!

[irock]

Okay. I almost found what I need.

I searched around and found a script, but it is not available for download. I need to retreive this for tweaking.

Could anyone pls help?

[edited by: jatar_k at 8:40 pm (utc) on Feb. 5, 2005]
[edit reason] removed url [/edit]

[jatar_k]

what exactly are you trying to do irock?

is it just to check the site? or are you wanting to backup the html with it?

if you are strictly looking for monitoring there are many other programs you could use or you could even quickly write your own.

The short cron might backfire as that is a very small interval, if it doesn't finish then you could have some very unexpected results.