Vix - Nif bindings for vips

akash-akya · November 28, 2020, 3:23pm

Hi everyone,

I have been hacking on NIF bindings for vips for sometime to understand NIF, macros and C better. libvips is a demand-driven, horizontally threaded image processing library. Compared to similar libraries such as imagemagik, libvips runs quickly and uses little memory.

Binding for vips operations are generated using GObject introspection, so documentation and bindings are up-to-date with the vips version installed. It uses the dirty scheduler to avoid blocking main schedulers.

Check vix operations documentation for the list of available operations and spec.

Vix also contains a very simple implementation to interact with GObject, which might be useful for other GObject based bindings such as GStreamer. Maybe it should be moved to a separate library in the future.

Github

Example

alias Vix.Vips.Image
alias Vix.Vips.Operation

def example(path) do
  {:ok, im} = Image.new_from_file(path)

  # put im at position (100, 100) in a 3000 x 3000 pixel image,
  # make the other pixels in the image by mirroring im up / down /
  # left / right, see
  # https://libvips.github.io/libvips/API/current/libvips-conversion.html#vips-embed
  {:ok, im} = Operation.embed(im, 100, 100, 3000, 3000, extend: :VIPS_EXTEND_MIRROR)

  # multiply the green (middle) band by 2, leave the other two alone
  {:ok, im} = Operation.linear(im, [1, 2, 1], [0])

  # make an image from an array constant, convolve with it
  {:ok, mask} =
    Image.new_matrix_from_array(3, 3,
      [
        [-1, -1, -1],
        [-1, 16, -1],
        [-1, -1, -1]
      ],
      scale: 8
    )

  {:ok, im} = Operation.conv(im, mask, precision: :VIPS_PRECISION_INTEGER)

  # finally, write the result back to a file on disk
  :ok = Vix.Vips.Image.write_to_file(im, "out.jpg")
end

Simple unscientific comparison with mogrify (bindings for imagemagik)

For generating thumbnail

	Vix	Mogrify
1	298.731ms	618.854ms
2	29.873ms	605.824ms
3	34.479ms	609.820ms
4	31.339ms	604.712ms
5	32.526ms	605.553ms

Notice that the gain is significant for the subsequent calls to the operation since vips caches the operations.

Warning

This library is experimental, untested, and unstable. Interface might change significantly in the future versions. The code is not well tested or optimized, so you might experience crashes.

Please check project readme for more details.

vincer · November 28, 2020, 9:26pm

Very useful library, looking forward to seeing it progress. I’m currently using the sharp library for this (https://github.com/lovell/sharp) with Porcelain. I’ll try this out in a non-production setting.

akash-akya · November 29, 2020, 6:04am

Thanks, let me know if you face any issues.
Next important feature is supporting VipsConnection, but it’s a little tricky and might take some time.

akash-akya · June 12, 2022, 11:12am

Updating progress after a long time. With the new release, Vix is at v0.10.0 now. It has few significant additions, which I think worth sharing.

v0.10.0

Experimental support for streaming. User can read or write images without keeping the complete image in memory. See Vix.Vips.Image.new_from_enum/1 and Vix.Vips.Image.write_to_stream/2
Efficient interoperability (zero-copy) with other libraries such as Nx, eVision. See Vix.Vips.Image.new_from_binary/5 and Vix.Vips.Image.write_to_tensor/1

See release changelog for more details.

Other changes include support for foreign operations, copying as formatted binary, updating image metadata etc. With this, almost all major libvips functionality can be accessed in Elixir (If you think something is missing, please let me know).

I like to thank @kip for his recent contributions to Vix and encouragement

Should I use Vix or Image?

Vix tries to be close to libvips interface to support a large set of use cases. Because of this, doing some basic image operation might feel unintuitive. Image by @kip is an excellent library, which builds on top of Vix and provides more elixir friendly wrapper functions for common operations along with many additional features such handling Exif, Math operators, and more. And all of this is accompanied by good documentation. So for most of the users, using Vix via Image might be better choice.

akash-akya · February 3, 2023, 2:27pm

Hi all, sharing update since there are major changes since the last update.

Vix now supports prebuilt NIF and libvips binaries. This should solve all the hiccups users used to face when using Vix. Issues such as installing correct libvips version, compiler tooling, libvips dependencies, compiler flags etc. With release v0.16.0 you don’t have to install anything, just add {vix, "0.16.0"} and forget about it. This should also make using Vix in Livebook or script more straight forward. Pre-built binaries are managed using cc_precompiler. This is a breaking change if you depend on an operation which is not present in the prebuilt binary, please check release docs for more details.

Other major changes

Image now implement Access behavior in a manner similar Nx.Tensor. You can use it to access pixel data from in a more intuitive way.

{:ok, img} = Image.new_from_file("./test/images/puppies.jpg")

# Extract an image band
red = img[0]

# Slices the top left 10x10 pixels of the image
# and returns all bands.
top_left = img[[0..9, 0..9,]]

# Returns an image that slices a 10x10 pixel square
# from the top left of the image with three bands
area = img[[0..9, 0..9, 0..3]]

# Slices the bottom right 10x10 pixels of the image
# and returns all bands.
bottom_right = img[[width: -10..-1, height: -10..-1]]

See document for more details. Thanks to @kip for this contribution.

Vix Image now implements Kino.Render protocol, so we can use it livebook with image preview out of the box.

Thanks to all contributors and open source libraries on which Vix is based on.

@kip for multiple contributions
prebuilt libvips binaries is based on excellent work being done on Sharp: GitHub - lovell/sharp-libvips: Packaging scripts to prebuild libvips and its dependencies - you're probably looking for https://github.com/lovell/sharp
cc_precompiler by @cocoa

kip · February 3, 2023, 2:35pm

Congratulations @akash-akya, a great update and definitely excited to use this especially with Livebook. Been working with Vix for over a year now and never had a segfault or a memory leak, just great performance, low memory usage and good fun.

cjbottaro · February 5, 2023, 8:48pm

I use Vix via @kip’s Image library. So far so good!