I've been running a php script for years that regularly grabs content from a set of URLs (10 at a time), and processes their content. I use a fairly standard implementation of the CURL_MULTI approach, and all is well. Except when it isn't...
The problem comes in with removing handles from the multi-handle (this is required if you want to reuse the multi-handle with new URLs). For some reason, my script often just stops dead at that point. I'm fairly certain that it's because of excessive CPU cycle usage during this operation, which causes my shared server to terminate the php script. It's beyond me why this operation should be CPU-hungry, but I do see (when debugging) that each handle removal takes an appreciable amount of time.
I know it's a long shot, but perhaps some of you have run into the same issues. Any help would be much appreciated...
Rob
for ($i=0;$i<$chc;$i++) {
curl_setopt($chh[$i],CURLOPT_URL,$chu[$i]);
curl_multi_add_handle($mh,$chh[$i]);
}
do {$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active and $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do { $mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
(....content processing happens here....)
for ($i=0;$i<$chc;$i++) {
curl_multi_remove_handle($mh,$chh[$i]);
}
(....the problem occurs in this last line - not always, and not always at the same value of $i....)