image processing

python, php, imagick, pil

[belfasttim]

I'm curious if anyone has information or experience on this--

I have a large number of images to process (create thumbnails, etc.), around 50,000 a week or so. I have written a PHP script that uses imagick for this process, but it's been a little sketchy with memory issues.

I am starting to rewrite the script in python using the python image library (PIL) to do the convert.

Does anyone know from experience whether this is likely to make a difference in terms of speed or memory usage (before I waste too much time on it)?

Thanks!

[rocknbil]

You shouldn't have memory problems with imagick/imagemagick, that's one of the reasons for using it over the GD toolkit. ImageMagick launches it's own process separate from PHP and doesn't use as much memory because it performs a lot of it's operations in virtual memory on disk (not that I "know" this, it's how it was explained to me by someone who does.)

If you're having memory issues, maybe it's the way this task is being performed. How are you doing it now, via cron job, HTTP request, PHP script? What's the environment, strong server, dedi/VPS, or . . . ugh . . . shared hosting?

Is it because it's launching too many processes at once? I'm visualizing parsing through a database, and launching an exec command 50,000 times, creating 50,000 processes. That would do it . . . when instead, make it linear, completing each resize before moving on to the next thumbnail, let it take a little longer. (Am I close?)

50,000 is not that much, really, it should be able to whip through 50K image resizes in a couple hours.