Does Image Rendering "shell out"?

fireproofsocks

I’m hoping an Erlang/Elixir Jedi can help show me the ways of the force here. I got a bunch of images that need to be manipulated programmatically (stop me if you’ve heard this before). So I’m looking over the approaches written about here:
https://sergiotapia.me/generate-images-with-name-initials-using-elixir-and-imagemagick-374eca4d14ff
and here Elixir Native Interoperability – Ports vs. NIFs

Basically, the question comes down to this: is the image manipulation process “booting up” each time an image is sent to it for rendering? Or is there a way to boot up the image manipulating code (whatever it is) ONCE when the Elixir app starts, and then run each image through it?

In other languages, any time we “shell out” to an ImageMagick or similar command, the Image processing utility starts up each time the call is made. I’m imagining this like starting up Photoshop on my desktop: you launch the app, you see the banner image, the spinning wheel, the app loads, then the image editing occurs, and then it shuts down. Wash, rinse, repeat. However, we are crunching through millions of images, so that boot-up time ends up being prohibitively expensive (both in terms of rendering time and CPU).

Can someone help explain the differences between the different approaches available to Elixir? System.cmd() seems to represent “shelling out” – i.e. whatever it is calling, that executable needs to load up each time the command is called. Whereas NIFs and ports seem to be able to boot up once and stay running alongside the parent Elixir application. Is that accurate? And there might be some Elixir module we can use? Given the need for this process to be as performant as possible (and not reload executables redundantly or blow up our EC2 bill), what is the best way to accomplish a task like this?

(FYI: the image transformations are relatively simple: mostly flattening layers or doing simple resizing). Others have expressed concern about using Erlang for computationally-heavy operations, but I would like to at least give it a good honest benchmarking. If not Elixir/Erlang, C or Go or Rust (or ???) might be preferable solutions for handling high concurrency and being able to natively perform the image manipulations without “shelling out”.

Thoughts or suggestions welcome. Thanks!

3 comments

#processes #image-processing #threads

1 1349 3

2018-06-06 14:42:43 UTC

Showing Posts 1 to 3

Show Best Posts
Show All (oldest first)
Show All (newest first)

NobbZ

I only glanced over the blog post, but yes, it is “shelling out”.

But to my experience so far, IMs boot up time is neglible.

If you really want to, you can of course try to interface the C++ library in a NIF or C-Port, but I am not sure if this will make a huge difference…

Post #1

dimitarvp

What you can do is always have 1 or more executables started via System.cmd and waiting on their stdio for data. That’s a trick I used in other programming languages when I was concerned about the initial overhead of loading a native executable in a child shell; because now the executable is loaded and is only waiting for data.

One example coming to mind was re-coding videos through ffmpeg; I’d start it with all necessary flags but without any input and it would just wait on stdio until I send it a video to process. Once sent, it will process it immediately.

Can’t give you exact details but that’s usually how a pool of disposable native processes is managed.

Post #2