Forum Moderators: coopster

Message Too Old, No Replies

Curl-multi CPU timeout problems

         

RobBroekhuis

1:49 pm on Aug 8, 2008 (gmt 0)

10+ Year Member



I've been running a php script for years that regularly grabs content from a set of URLs (10 at a time), and processes their content. I use a fairly standard implementation of the CURL_MULTI approach, and all is well. Except when it isn't...
The problem comes in with removing handles from the multi-handle (this is required if you want to reuse the multi-handle with new URLs). For some reason, my script often just stops dead at that point. I'm fairly certain that it's because of excessive CPU cycle usage during this operation, which causes my shared server to terminate the php script. It's beyond me why this operation should be CPU-hungry, but I do see (when debugging) that each handle removal takes an appreciable amount of time.
I know it's a long shot, but perhaps some of you have run into the same issues. Any help would be much appreciated...
Rob

for ($i=0;$i<$chc;$i++) {
curl_setopt($chh[$i],CURLOPT_URL,$chu[$i]);
curl_multi_add_handle($mh,$chh[$i]);
}
do {$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active and $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do { $mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
(....content processing happens here....)
for ($i=0;$i<$chc;$i++) {
curl_multi_remove_handle($mh,$chh[$i]);
}
(....the problem occurs in this last line - not always, and not always at the same value of $i....)

eelixduppy

2:04 pm on Aug 8, 2008 (gmt 0)



The first thing I'm going to ask for is for you to give us any errors that you may be getting from this script. This is gonig to be the first step in finding a solution if there is one.

RobBroekhuis

2:24 pm on Aug 8, 2008 (gmt 0)

10+ Year Member



No php errors (even low-level) are generated - just a sudden-death exit from the script, probably initiated by server processes rather than php itself.

[edited by: RobBroekhuis at 2:25 pm (utc) on Aug. 8, 2008]