Online Forever?

can i create a cgi to be online and do something

[Roozbeh]

Hi

can i make a program and then upload it to my site and so it will be online forever?...

for example i want to write an application which will gather some information from some sites every minute and save them into my site...

or making a search engine..ie search robot...to gather informtation from sites...

i dont want this program become active by browsing or...just being always active...

Thanks
Roozbeh

[divaone]

do a search for 'cron' and 'crontab'. its a file you put on your server with commands to do something every so often. your host needs to allow the execution of crontab for it to work successfully. if it does not, search for a script that will simulate cron. usually tho, these scripts rely on someone activating something, like entering your site and viewing (loading) an image.

good luck

[Zaphod Beeblebrox]

If you have physical access to your server, you could simply install a program and run it.

I have one running that regularly deletes spam messages from my mailserver's inbox directories, updates my stats database with reverse DNS lookups and stuff, and checks my webbased agenda to see if there are any reminders to be emailed...

[bakedjake]

You could write a daemon or service, if your host will let you run executables on their local system under your user account.

[hanuman]

you got correct advise. only to add that you should consult your ISP if you are on share host, your utility might consume too much cpu for you to be friendly neighbor on the server. than you should considering getting your own box....

[stargeek]

I've written a spider and had problems with cron on my server
so i wrote a command lind php script that waits for a certain number of seconds then runs a program using exec(); you then run the php script at the command line as a background process like (php script.php &) this is a little hacked up but its worked great for a year for me.

[Roozbeh]

can you give me your spider?
i mean i want somewhere to begin i know nothing of site development and is very newbie in this field and was just wondering what can i do.....

any site that got some usefull resources?

also becouse of spams my yahoo mail is become full everyday is there any script which can free my bulk so i can start it every day......

also i see cron in my cpanel and it runs command i geuss.
can i write application and upload it and make it by cron run every minute?also if i make application not to end ie making it work in a loop so i am online forever i guess...true?

[Avitar]

First you shouldn't need to call the program every few minutes to run since that will make a new instance of the program every few minutes. You should just run the program as a Daemon or Service if possible.

Be aware that this sort of program will suck you bandwidth up quickly.

In addition many sites and/or servers will detect a spider. If your spider does not head their robots.txt file and other meta directives your IP will get banned.

In addition there are many copyright issues you need to consider if you do not want to be shut down with a lawsuit.

Be aware that 'linking' to a site requires legal permission, though several webmasters may be kind about it.

It is for these reasons I suggest you instead spend your time tracking down good RSS feeds. There are many free/cheap content providers that provide XML feeds from their site.

If you insist on a spider, try looking up search engine code for starters.

Good Luck

-----
David Smith