Multiple processes to check progress

[CodilX]

Hello,

I can't get this darn thing to work...

I have a script that imports values from a .txt file and uploads them to MySQL.

I'm trying to create a sort of progress bar for this.

I have process.php, that goes through every line from a .txt file, adds the value to MySQL and adds +1 to $_SESSION['progress']

Then I have a progress.php, that echos the $_SESSION['progress'] to see how many values are imported.

I'm trying to add a loop with javascript, that checks the progress.php file and with jQuery I add the number to a div to track progress. The loop itself works fine, but I can't even access progress.php from the browser when process.php is working.

On average I have around 150,000 of lines in the .txt files, and it takes a few minutes sometimes to add the values into MySQL, because it has to regexp to check if the value is valid.

Maybe I'm trying the wrong approach, but it's very important to see the progress of the process.php while it's importing values to MySQL.

Can someone help me out with this?..

Thanks!

[Wittner]

Have you thought to use jQuery/AJAX to import the text file? Then you could use jQuery's built in support for the progress showing either a progress bar or cycling circle as your loading image. If you have control over your .txt file, you could make it a JSON text file which jQuery can use natively, also very quick! - just some thoughts on another approach. Your approach seems instinctively a little complicated to me.

[CodilX]

That seems like an idea :) but I have no idea how to implement it, as I just started using jQuery. Do you have an example? Thanks :)

[CodilX]

OK I managed to get this to work:

js:

$.getJSON('add.php?do_file=json',function(data) {
var total;
var pixels;
var percent;
$.each(data, function(i,data) {
if(i == 0) {
total = parseInt(data);
} else {
$.get('add.php', { email: data }, function(response) {
percent = i*100/total;
pixels = 400*percent/100;
$('.progress').css({ 'width' : pixels + 'px' });
$('.percent').text(Math.ceil(percent) + '%');
});

}
});
});

The problem is that it puts massive strain on the PC. When I try loading a 3mb .txt file my PC goes insane..

[Wittner]

I didn't think about size I'm afraid. Importing that many files to a MySQL database would be time consuming anyhow and maybe that's too large for AJAX. I guess your pc memory is being gobbled up by the script, although 3mb doesn't sound too massive to me. Is this import scipt just for yourself or a web service you are making available to people via their browsers?

[CodilX]

The service will be used by a few people, but it would be nice of course having it as fast as possible. Especially it would be annoying waiting for 15+ minutes to import the values from the .txt file, when you could rewrite the .txt file to actualy MySQL insert queries and run them, which wouldn't take as long. But try explaining it to people who don't want to use phpmyadmin or similar :)

I modified the code once more. Now the php script has a $_SESSION['added'] as an index for which line from the .txt has been read. So it reads line 1, adds +1, and the next time the .php is called, it reads line 2 and so on. The js looks like this:

var total;
var i = 1;
$.get('add.php?count=emails', function(data) {
total = data;
function addValue() {
$.get('add.php?value=add', function(response) {
if(response == 1) {
percent = i*100/total;
pixels = 400*percent/100;
$('.progress').css({ 'width' : pixels + 'px' });
$('.percent').text(Math.ceil(percent) + '%');
i++;
addValue();
} else {
$.get('add.php?delete=txt', function(data) {
$("#load_file").text('Values imported.');
});
}
});
}
addValue();
});

It works just fine, no lag, no strain, as it doesn't add 150,000 json entries to the users cache :) BUT it relies on the users internet connection, because the js loop has to call the .php file for every line. When I'm using localhost, the import finished, while the same import initiated at the same time was at 23%. So just because the import relies on the users internet connection, it import ~4 times slower.

I'm still thinking of going back to using a one process php file, and some how tracking it's progress, because having the import script relying on the users internet connection isn't a solution..

Any ideas? :/

[Wittner]

just thinking out loud... is there any way to NOT use the users' browsers for the upload. I realise how daft that sounds, but could they somehow just initialise the upload but have it happen from server to server and maybe they get an email back when it's finished.... not even sure I understand what I'm saying myself!

cheers,

Wittner

[rocknbil]

What you are likely going to need here is a fork. I've done these before, it's a brain burner, but the basic concept is like this.

You have this time intensive process and it often times out in the browser. What you want is to lunch this process as a background process, then monitor it's progress.

The problem with web apps is even with sessions, which maintain their connection to the client with the PHPSESSID cookie, is stateless. We send a request, the process dies. If it's a large process, the browser may time out and the process may continue, but we just don't know if it does or it doesn't.

Enter fork:

if ($pid = fork()) {
// I AM THE PARENT
}
else {
// LAUNCH CHILD PROCESS HERE
}

(This is perl code, PHP has an equivalent)

The child process contains your time intensive process identified by the process id in $pid. This allows your parent process to return an immediate response to the browser which child works away. When the child process dies a natural death, it's done.

A good use of the parent is to output a frame containing an http refresh that queries for the existence of $pid, and if it does executes a count on the database:

select count from database . . .

echo "$count entries updated . . . "

(else output "we're done")

There are many was to go with it, but fork() is a handy one.

[CodilX]

Hmm.. I've just read some tuts and info about fork(), but I'm not sure if that's what I want :/ Maybe I'm wrong? I've never heard of this function, maybe someone could shed some light on it? As I understand, the parent process will return AFTER the child process is finished? What I want is constant information on how many values have been imported.

The thing is.. This percentage thing is really necessary, even using a fast server, reading a file with 150 000 and sometimes more lines of values, parsing it, checking if the values are valid, and importing them into MySQL - takes a hell of a long time by all standards. So it's very nice to see how far along the process is at a given time. All day I'm thinking of different ways to track the progress, but I just can't come up with a way that won't involve the users browser in a way that it can affect the import itself. I wouldn't mind if the progress would be updated every 5 or more seconds, I don't need it to constantly update, just to show the user using the system, that something IS happening and nothing stalled or something.

My .php file, or the main part of a loop, after the file was uploaded to the server, is very simple:

$file = file($_SESSION['file']);
regexp_if_valid($file[$_SESSION['index']]);
if valid {
 add_to_mysql($file[$_SESSION['index']]);
}
$_SESSION['index']++;

I'm wondering if it's worth trying to add the .php process using the exec() function, so it would work in the background, but I believe $_SESSION won't work right? I could show what file to parse by adding a $_GET variable to the .php file, and a flat file for holding the index value, BUT opening, reading, writing and closing the file for each value also doesn't feel right :/

I dunno.. All day I'm trying everything I can think of.. Darn PHP.. If only it could run a process file in the background without making the system unusable at that moment..

[rocknbil]

Hmm well, yeah. You spawn a child and fork returns it's process id, allowing the child to chomp away in the background and freeing the parent to return a response. pcntl_fork() [us.php.net].