Image - an image processing library based upon Vix

[kip]

Image is an image processing library for Elixir. It is based upon the fabulous vix library that provides a libvips wrapper for Elixir.

Image is intended to provide well-documented, performant and reliable common image processing functions in an idiomatic Elixir functional style. It operates at a layer above the very comprehensive set of functions in Vix and libvips.

In a very simple image resizing benchmark, Image is approximately 2 to 3 times faster than Mogrify and uses about 5 times less memory.

Since Image is based upon Vix and libvips it is performant, concurrent, pipelining and has a low memory footprint.

In this first release it focuses on resizing, cropping, masking, corner rounding, circular cropping, metadata extract, metadata minimisation. It also includes some simple functions to make it easy to resize and compress images for many well-known social media platforms at the correct size.

In the next two releases, Image will:

• Provide streamed image processing. That will allow an image to be streamed from a file, or from S3 or from any Elixir stream or enumerable, process the image and then stream its output - including to chunked responses for HTTP applications.
• Provide bi-directional Integration with Nx that will efficiently share memory buffers and make it even simpler to involve image processing in ML applications.

Simple examples

Resize to fit

Image.resize(image, 200, crop: :none)

Resize to fill

Image.resize(image, 200, crop: :attention)

Crop image

Image.crop!(image, 550, 320, 200, 200)

Rounded corners

image |> Image.resize!(200, crop: :attention) |> Image.rounded!()

Avatar (circular mask, remove most metadata, crop to a subject of interest)

Image.avatar(image, 200)

[webuhu]

Wonderful.
Mind to tag within the repository for the ex_doc source links and links on Hex.

[kip]

Whoops, sorry about that. Done!

[jeroenvisser101]

Hey Kip, this is awesome! In addition to streaming, to ‘streamline’ the API, will you also add functions to load images from Memory, and will the S3-thing be a behaviour so other backends (such as GCS) can be added as well?

PS. Have you considered applying for GitHub Sponsors? I would love to sponsor you there as I use many (or even all) of your libraries, and would love to help even if in a small way! If you do, let me know

[mayel]

This looks great! I’ve been using imagemagick via Mogrify so far and while I’d like to use vips, I would prefer having the option to shell out rather than using the NIF when dealing with user-uploaded images (for the reasons you point out in your readme). Is this something you would consider adding to the library?

[AndrewDryga]

@kip does it support overlays (drawing an image on top of another one), transparency, and writing a text on an image?

[Exadra37]

@AndrewDryga

overlays (drawing an image on top of another one), transparency, and writing a text on an image?

Something like shown in the animation in the home page of:

imgix - The End-to-End Visual Media Solution

[kip]

@AndrewDryga

does it support overlays (drawing an image on top of another one), transparency, and writing a text on an image?

vix and libvips support everything you can think of, possibly even making you coffee in the morning. Text overlays are definitely on the list of features to expose. I’ve been working on what the right API should look like - feedback welcome. In the Vix readme there is an example of rendering a text overlay (see below).

# render text as image
# see https://libvips.github.io/libvips/API/current/libvips-create.html#vips-text for more details
{:ok, {text, _}} = Operation.text(~s(<b>Vix</b> is <span foreground="red">awesome!</span>), dpi: 300, rgba: true)
# add text to an image
{:ok, img_with_text} = Operation.composite2(img, text, :VIPS_BLEND_MODE_OVER, x: 50, y: 20)

@mayel

I would prefer having the option to shell out rather than using the NIF

@mayel, I understand the concern, and this is of course a benefit of using mogrify. Nevertheless, shelling out doesn’t resolve the issue of maliciously crafted images, and there is a long history of CVEs. So while you avoid a crash in mogrify bringing down the BEAM, there are other issues. Its a tradeoff. Image, which depends on Vix, is firmly wedded to libvips as a NIF. I am going to try some experiments in running Image in its own BEAM node but that’s as far as it will go. I have to note that so far the stability has been excellent - only crashing when I made a poorly written addition to the NIF (which @akash-akya promptly fixed).

[kip]

@jeroenvisser101, the streaming API will do exactly what you want. @akash-akya has an experimental branch with the NIF implementation that basically means that image streaming can consume any Enumerable and as an output can produce an Enumerable.

This is fabulous for taking an S3 image, transforming it, and sending it as a chunked image in an HTTP request. There are many other use cases, but I think that one is high on the list of valuable capabilities.

Which is why it will be part of the Image 0.2 release in a week or so.

If you’re interested you can take a look at these tests in Vix.

[kip]

A couple of addition thoughts on the overall imaging pipeline strategy. What appeals to me about a library based upon libvips is that it is fast, has a small memory footprint and has really good concurrency support. These alone are for me enough to accept the potential issues with a NIF-based solution.

When you also have:

• An architecture in libvips that is very friendly for a NIF implementation. This because only references are ever passed from Elixir-land to NIF-land so no large memory copies.
• An internal streaming architecture in libvips that means transformation evaluation is only done at the point at which an image is rendered (either by writing to a file or sending to a client for example). Its very efficient.
• Really strong support from @jcupitt, who is active, engaged and responsive
• The ability to do all the transformation in memory, with streamed data

Then I think libvips with Vix is a really strong platform - as long as the concerns relevant to any NIF-based solution are recognised. Its analogous a little bit to Nx, the benefits outweigh the risks for a number of use cases.

[AndrewDryga]

We are processing a ton of images with internal service that works on top of graphicsmagick but we faced many issues with it (like it can corrupt its internal miff format during text-based transforms). We have a workaround for those but libvips is also an option so I was asking if the library supports that, so maybe we can run an experiment replacing gm with vix. We don’t need external services.

[kip]

@AndrewDryga sounds very interesting, and I’d be happy to collaborate on that is its useful to you. I have a similar motivation - next up in the list is another library I’m calling Imagine that is basically a plug to serve images using Image that can be very easily added to any Plug or Phoenix application. A sort of a mix of thumbor and Cloudinary. But dead simple to add to an Elixir application and really easy to use.

Imagine is also going to make strong use of Image Client Hints that I think is the next big breakthrough in efficiency of web images.

Oh, and of course since you’re already knee-deep in image processing, using Vix is a great idea, its fabulous. It takes a bit of effort to bridge between libvips terminology, the Vix implementation and your requirements but its not difficult.

[Exadra37]

@AndrewDryga

. We don’t need external services.

I was not suggesting to use external services, just trying to exemplify with the example I gave if that was what you were looking for.

[akash-akya]

@mayel

I would prefer having the option to shell out rather than using the NIF when dealing with user-uploaded images

Apart from what @kip already mentioned, vips command currently only allows one operation per execution. So if we need to run more than one operation, we have to run multiple commands. This miss out on almost all the optimizations and performance gain libvips provides, I think performance is the reason people switch to libvips from ImageMagick. There is also reusing the intermediate result aspect. With shelling out, we can’t do any of that.

But if you only have one operation, and you are okay with trading performance for safety, it’s straight forward to wrap vips command. We can even stream results using libraries which support streaming commands. Like ex_cmd or exile [Disclaimer: I’m the author of these two libraries]

input = File.stream!("foo.jpg")

# takes stream as input and returns stream as output
output = ExCmd.stream!(~w(vips invert stdin .png), input: input)

Stream.into(output, File.stream!("inverted_foo.png"))
|> Stream.run()

You can spawn shell and pipe output across commands too

cmd = "vips invert stdin .png | vips gaussblur stdin .png 1.5"
output = ExCmd.stream!(["sh", "-c", cmd), input: input)

---

will you also add functions to load images from Memory, and will the S3-thing be a behaviour so other backends (such as GCS) can be added as well

@jeroenvisser101 If you are okay with shelling out and if your pipeline is simple, you can already achieve this like I mentioned above. But of course, you would miss out on all the features and heavy lifting Image library brings.

Disclaimer: I’m the author of Vix

[mayel]

Thanks for your response @akash-akya, that was helpful

[outlog]

great work (as always!) @kip - and especially @akash-akya for the vix library…

we should also see about upstreaming first class support to waffle, I’m currently using waffle and vix using this one-line patch ExOptimizer Integration? · Issue #76 · elixir-waffle/waffle · GitHub

defp apply_transformation(file, {:custom_func, func}), do: func.(file)

notice how pngs stay pngs and jpgs stay jpgs, also experimenting with pdf thumbnails - but I believe waffle really wants to know the file extension as in their convert api (notice last :png atom):
{:convert, "-strip -thumbnail 250x250^ -gravity center -extent 250x250 -format png", :png}

great performance benefits with vix

defp generate_thumb(file, width) do
    case Vix.Vips.Foreign.find_load(file.path) do
      {:ok, "VipsForeignLoadPng" <> _kind} ->
        {:ok, resized_img} = Vix.Vips.Operation.thumbnail(file.path, width, size: :VIPS_SIZE_DOWN)

        new_path = Waffle.File.generate_temporary_path(".png")

        case Vix.Vips.Operation.pngsave!(resized_img, new_path,
               strip: true,
               palette: true,
               effort: 10,
               bitdepth: 8,
               compression: 9,
               Q: 100
             ) do
          :ok ->
            new_file = %Waffle.File{file | path: new_path, is_tempfile?: true}
            {:ok, new_file}

          _ ->
            {:error, "It wasn't possible to generate a thumbnail"}
        end

      {:ok, "VipsForeignLoadJpeg" <> _kind} ->
        {:ok, resized_img} = Vix.Vips.Operation.thumbnail(file.path, width, size: :VIPS_SIZE_DOWN)
        new_path = Waffle.File.generate_temporary_path(".jpg")

        case Vix.Vips.Operation.jpegsave!(resized_img, new_path,
               strip: true,
               "optimize-coding": true,
               Q: 85,
               interlace: false,
               "optimize-scans": true
             ) do
          :ok ->
            new_file = %Waffle.File{file | path: new_path, is_tempfile?: true}
            {:ok, new_file}

          _ ->
            {:error, "It wasn't possible to generate a thumbnail"}
        end

      {:ok, "VipsForeignLoadPdf" <> _kind} ->
        {:ok, resized_img} = Vix.Vips.Operation.thumbnail(file.path, width, size: :VIPS_SIZE_DOWN)
        new_path = Waffle.File.generate_temporary_path(".jpg")

        case Vix.Vips.Operation.jpegsave!(resized_img, new_path,
               strip: true,
               "optimize-coding": true,
               Q: 85,
               interlace: false,
               "optimize-scans": true
             ) do
          :ok ->
            new_file = %Waffle.File{file | path: new_path, is_tempfile?: true}
            {:ok, new_file}

          _ ->
            {:error, "It wasn't possible to generate a thumbnail"}
        end

      unknown_loader ->
        IO.inspect(unknown_loader)
        {:error, "unknown format - only png/jpg supported"}
    end
  end

[kip]

@Exadra37 well, I couldn’t help myself so I put some work into an API for text and shape overlays. Not quite ready for formal release yet but its available to try from GitHub in the text branch. These demos are in the lib/demo.ex file.

Reproducing the example

The code that produced this is:

  def demo1 do
    {:ok, base_image} = Image.open("test/support/images/Singapore-2016-09-5887.jpg")
    {:ok, polygon} = Shape.polygon(@points, fill_color:  @polygon_color, stroke_color: "none", height: Image.height(base_image), opacity: 0.8)
    {:ok, explore_new} = Text.new_from_string("EXPLORE NEW", font_size: 95, font: "DIN Alternate")
    {:ok, places} = Text.new_from_string("PLACES", font_size: 95, font: "DIN Alternate")
    {:ok, blowout} = Text.new_from_string("BLOWOUT SINGAPORE SALE", font_size: 40, font: "DIN Alternate")
    {:ok, start_saving} = Text.new_from_string("START SAVING", font_size: 30, padding: 20, background_fill_color: "none", background_stroke_color: "white", background_stroke_width: 5)

    base_image
    |> Image.compose!(polygon, x: :middle, y: :top)
    |> Image.compose!(explore_new, x: 260, y: 200)
    |> Image.compose!(places, x: 260, y: 260)
    |> Image.compose!(blowout, x: 260, y: 340)
    |> Image.compose!(start_saving, x: 260, y: 400)
    |> Image.write!("/Users/kip/Desktop/polygon.png")
  end

Simple Text Overlay

Which is created with:

  def demo3 do
    {:ok, base_image} = Image.open("test/support/images/Singapore-2016-09-5887.jpg")
    {:ok, singapore} = Text.new_from_string("Singapore", font_size: 100, font: "DIN Alternate")

    base_image
    |> Image.compose!(singapore, x: :center, y: :middle)
    |> Image.write!("/Users/kip/Desktop/center_text.png")
  end

Transparent Text

This one is quite fun. Reversed text, full screen overlay with transparency. Created with:

  def demo2 do
    {:ok, base_image} = Image.open("test/support/images/Singapore-2016-09-5887.jpg")
    {:ok, singapore} = Text.new_from_string("SINGAPORE", font_size: 250, font: "DIN Alternate", padding: base_image, text_fill_color: :transparent, background_fill_color: "black", background_fill_opacity: 0.6)

    base_image
    |> Image.compose!(singapore)
    |> Image.write!("/Users/kip/Desktop/overlay.png")
  end

Should be able to have both text overlays and shape overlays finished on the weekend.

[stevensonmt]

I really love how straight forward the code looks to achieve the somewhat complex layered results. So nice!

[kip]

Thanks for the feedback! If you liked that, here’s the version that uses relative positioning which is much more intuitive to use and produces the same image as the first example above:

  def demo4 do
    ....

    base_image
    |> compose!([
      {polygon, x: 250, y: 0},
      {explore_new, y_baseline: :top, x_baseline: :left, dx: 20, dy: 200},
      {places, dy: 10},
      {blowout, dy: 20},
      {start_saving, dy: 50}
    ])
  end

[stevensonmt]

Very nice, though I actually think I prefer the almost builder-pattern approach of piping each layer to the next function. Makes it very clear how the image is being built.