How do I queue and execute processes?

[usrbin]

I'm trying to process incoming file requests, however I need to throttle them to execute the requests one at a time (or X at a time in the future).

Each incoming file request executes a PHP script and I'd like to email the user with the file as soon as its ready.

I'm thinking about having the incoming request script append a txt document with the requested file and email address, then "alert" another script that theres something on the list.

That processing script could grab and remove the top request off the document, create the requested file, and email it to the user. After it could check the document to see if there is anything else and processing the top item, or stopping if the list is empty.

I could create a cronjob to run a script that will check the document to make sure its empty and to start a processing script (if one, or the max, isn't already running) encase there were any hiccups.

Either way I'd want to make sure the script only runs 1 or x amount of times at any given moment and that users recieve the file as soon as possible. Is there a better way of doing this?

[Andy Langton]

If you don't want to introduce too much complexity, you can just have your script write to a file when it is active, and then delete the entry when complete. Any new instances can just exit if the file indicates a running process. Schedule your cron to run at a time which reflects how long a queue is likely to complete.

[usrbin]

That's a great idea, thank! With your feed what I'm thinking now is:

1 - Have the running request script check the value of instances.txt.
2 - If the value equals MAX then append queue.txt with request and stop.
3 - Else, increase value of instances.txt by 1 and process request.
4 - After processing request check queue.txt for any entries.
5 - If queue.txt is empty then decrease instances.txt and stop.
6 - Else remove request from queue.txt, process request, and go to line 4.

I'm wondering if simple txt documents are a good idea or if I should use a database.

I'm worried about scripts stopping due to some error, not processing the request, and not decreasing the instances.txt file. I'm also worried that of one script about to stop and just has to process line 5 while another script is in the middle of processing line 2.

[robzilla]

I would probably opt for a database instead of plain text files, so that you don't need to think about issues like concurrency in the future. Text files can be more complex, and more difficult to maintain. Simply insert each job into a table, and toggle the value of a "completed" column when processing has finished. A database also allows you to build a history more easily and efficiently, with other data such as processing state (important when you have multiple workers), time added, start time, end time, errors, etc. You could daemonize a PHP script that constantly checks the database for unprocessed jobs, begins processing when it finds one, or sleeps for a period of time when the queue is empty. Cron is limited to > minutes, whereas a PHP script can be put to sleep for an X number of seconds. A bash script (run through a cronjob) could check to make sure the PHP daemon (or any number of them) is active.