Image - an image processing library based upon Vix

Your Libraries & OS Mentoring Libraries
81

Motivated by the discussion on extracting frames from video, newly published Image version 0.22 supports some basic capabilities based upon the excellent Evision.VideoCapture module.

Enhancements

  • Adds Image.Video.image_from_video/2 to extract images from frames in a video file or video camera. Includes support for :frame and :millisecond seek options. Seek options are only supported for video files, not video streams.

  • Adds Image.Video.stream!/2 that returns an enumerable stream of frames as images. The stream takes a range as a parameter. For example:

  • Adds Image.Video.scrub/2 that scrubs the video head forward a number of frames.

  • Adds Image.Video.seek/2 to seek the video head to the requested frame or millisecond. Seeking is supported for video files only, not video streams. Seeking is not guaranteed to be frame accurate (due to underlying OpenCV issues).

Examples

# Extracting an image
iex> {:ok, video} = Image.Video.open "./test/support/video/video_sample.mp4"
iex> {:ok, _image} = Image.Video.image_from_video(video)
iex> {:ok, _image} = Image.Video.image_from_video(video, frame: 0)
iex> {:ok, _image} = Image.Video.image_from_video(video, millisecond: 1_000)

# Streaming images
# Extract every second frame starting at the
# first frame and ending at the last frame.
iex> "./test/support/video/video_sample.mp4"
...> |> Image.Video.stream!(frame: 0..-1//2)
...> |> Enum.to_list()
[
  %Vix.Vips.Image{ref: #Reference<0.2048151986.449445916.177398>},
  %Vix.Vips.Image{ref: #Reference<0.2048151986.449445916.177400>},
  %Vix.Vips.Image{ref: #Reference<0.2048151986.449445916.177402>},
  ...
]
5 Likes
82

@kip there is another regression straight from dependabot here:


== Compilation error in file lib/image/options/video.ex ==
Error: ** (CompileError) lib/image/options/video.ex:8: Evision.VideoCapture.__struct__/0 is undefined, cannot expand struct Evision.VideoCapture. Make sure the struct name is correct. If the struct name exists and is correct but it still cannot be found, you likely have cyclic module usage in your code

This is similar to the bumblebee stuff where you should safeguard the existence of dependent libraries and flag on/off your features I reckon?

83

Thereā€™s mix compile --no-optional-deps --warnings-as-errors to make sure that the application compiles successfully without optional dependencies present. Sounds like that would be useful to have in CI.

84

Arrggghhhh. Fixed in Image version 0.22.1. Thanks for the report of this authors idiocy!

1 Like
85

Yep, I certainly need to take some time to configure CI properly. Thatā€™s a chore I find hard to prioritise but youā€™re right.

1 Like
86

I just published Image version 0.23.0 with the following improvements and bug fixes. In particular Image.normalize/1 and Image.autolevel/1 are steps towards automated image improvement.

Bug Fixes

  • Fix specs for Image.Options.Write. Thanks to @jarrodmoldrich. Closes #36.

  • Fix spec for Image.exif/1. Thanks to @ntodd for the PR. Closes #35.

Enhancements

  • Adds Image.normalize/1 which normalizes an image by expanding the luminance of an image to cover the full dynamic range.

  • Adds Image.autolevel/1 which scales each band of an image to fit the full dynamic range. Unlike Image.normalize/1, each band is scaled separately.

  • Adds Image.erode/2 which erodes pixels from the edge of an image mask. This can be useful to remove a small amount of colour fringing around the edge of an image.

  • Adds Image.dilate/2 which dilates pixels from the edge of an image mask.

  • Adds Image.trim/2 which trims an image to the bounding box of the non-background area.

  • Adds Image.flatten/1 which flattens an alpha layer out of an image

  • Image.Options.Write.validate_options/2 now validates options appropriate to each image type in order to make validation more robust.

  • Adds :minimize_file_size option to Image.write/2 for JPEG and PNG files which if true will apply a range of techniques to minimize the size of the image file at the expense of time to save the image and potentially image quality.

12 Likes
87

Today is object detection Sunday. Greatly inspired by the fabulous talk by @hansihe at the Warsaw Elixir Meetup earlier this month I transcribed his live coding example and implemented it in a new experimental module, Image.Detection.

Based upon his solid observations I also added some quality of life improvements to the detect branch of Image:

  • Image.from_kino/2 and Image.from_kino!/2 to easily consume the image data from a Kino.Input.Image data source in Livebook
  • Image.Shape.rect/3 and Image.Shape.rect!/3 to have a composable way to draw rectangles - specifically object bounding boxes in this case.
  • Image.embed/4 and Image.embed!/4 to make it much easier to conform an image to the dimensions required by an ML model.

The code is ready for fun and experimentation. Given some of the tricky dependency configuration at the moment it can only be used as a GitHub dependency for now. You can add it in a mix.exs as:

{:image, github: "elixir-image/image", branch: "detect"}`

Demo example

Livebook coming this week!

iex> i = Image.open!("./test/support/images/elixir_warsaw_meetup.png")
%Vix.Vips.Image{ref: #Reference<0.3196308165.1633288229.90166>}
iex> Image.Detection.detect(i)
{:ok, %Vix.Vips.Image{ref: #Reference<0.3196308165.1633288229.90638>}}

elixir_warsaw_detected

The code

Its amazing how little code this takes with Nx, Axon and Axon Onnx.

  def detect(%Vimage{} = image, model_path \\ default_model_path()) do
    # Import the model and extract the
    # prediction function and its parameters.
    {model, params} = AxonOnnx.import(model_path)
    {_init_fn, predict_fn} = Axon.build(model, compiler: EXLA)

    # Flatten out any alpha band then resize the image
    # so the longest edge is the same as the model size,
    # then add a black border to expand the shorter dimension
    # so the overall image conforms to the model requirements.
    prepared_image =
      image
      |> Image.flatten!()
      |> Image.thumbnail!(@yolo_model_image_size)
      |> Image.embed!(@yolo_model_image_size, @yolo_model_image_size)

    # Move the image to Nx. This is nothing more
    # than moving a pointer under the covers
    # so its efficient. Then conform the data to
    # the shape and type required for the model.
    # Last we add an additional axis that represents
    # the batch (we use only a batch of 1).
    batch =
      prepared_image
      |> Image.to_nx!()
      |> Nx.transpose(axes: [2, 0, 1])
      |> Nx.as_type(:f32)
      |> Nx.divide(255)
      |> Nx.new_axis(0)

    # Run the prediction model, extract
    # the only batch that was sent
    # and transpose the axis back to
    # {width, height} layout for further
    # image processing.
    result =
      predict_fn.(params, batch)[0]
      |> Nx.transpose(axes: [1, 0])

    # Filter the data by certainty,
    # zip with the class names, draw
    # bounding boxes and labels and the
    # trim off the extra pixels we added
    # earlier to get back to the original
    # image shape.
    result
    |> Yolo.NMS.nms(0.5)
    |> Enum.zip(classes())
    |> draw_bbox_with_labels(prepared_image)
    |> Image.trim()
  end

Next steps

This is a proof-of-concept only. The API will almost certainly change - not all use cases require painting a bounding box with labels. Feedback however is most welcome!.

Thanks again to @hansihe, the work is all his.

15 Likes
88

Is there anywhere I can see a full script for this? I tried to get it running in Livebook and a newly created mix project but cant get the dependenies right :upside_down_face:

1 Like
89

Sure, itā€™s referenced in the post in the config line for mix.ibexā€™s and link behind the Image,Detection link. Just look at the detect branch at https://github.com/elixir-image/image`.

I had to do quite some munging to get the deps right too hence the GitHub dependencies.

1 Like
90

Iā€™ve made a simple Livebook to demonstrate how to install, configure and do object detection:

Run in Livebook

2 Likes
91

Is there an easy way to make a copy of a mutable Image? I looked around the docs for something, but I couldnā€™t find anything. Thatā€™s the reason I used the closure to make images for drawing in the talk.

1 Like
92

@hansihe, A mutable image is already copied for you and operations are serialised behind a genserver. So it is mutable - but only by you. And all operations are serialised so itā€™s also thread safe.

You can perform multiple mutations on a single copy of the image by using Image.mutuate/2 (I have just updated the documentation on Image.mutate/2 quite a bit to make this clearer).

Mutation example

# The image is copied and operations
# are serialized behind a genserver.
# Only one copy is made but all operations
# will be serialized behind a genserver.
# When the function returns the genserver
# is broken down and the underlying
# mutated `t:Vix.Vips.Image.t/0` is returned.

iex> Image.mutate image, fn mutable_image ->
...>  mutable_image
...>  |> Image.Draw.rect!(0, 0, 10, 10, color: :red)
...>  |> Image.Draw.rect!(2, 20, 10, 10, color: :green)
...>  |> Image.Draw.rect!(50, 50, 10, 10, color: :blue)
...> end

By wrapping multiple mutations in an Image.mutate/2 call there should be performance improvements - although for safety each mutation is still serialised through a genserver.

1 Like
93

Has anyone succeeded in pulling vix & image to a Windows machine? The docs says pre-build binaries are available but nothing is fetched.

94

Never tried on windows, but I didnā€™t have much luck on my Raspberry Pi either :slight_smile:

95

vix needs to be at least version 0.16.0 for prebuild binaries, and of course image depends on vix.

I donā€™t have any windows machines so Iā€™m afraid I donā€™t have much of an idea. Pinging @akash-akya to see if he has any suggestions.

1 Like
96

Thank you for this! Sharp mentions Windows binaries are available, so I wonder if itā€™s just a missing matching in cast_target of vixā€™s build_scripts\precompiler.exs

  defp cast_target(%{os: os, arch: arch, abi: abi}) do
    case {arch, os, abi} do
      {"x86_64", "linux", "musl"} ->
        {:ok, "linuxmusl-x64"}

      {"aarch64", "linux", "musl"} ->
        {:ok, "linuxmusl-arm64v8"}

      {"x86_64", "linux", "gnu"} ->
        {:ok, "linux-x64"}

      {"aarch64", "linux", "gnu"} ->
        {:ok, "linux-arm64v8"}

      {"x86_64", "apple", "darwin"} ->
        {:ok, "darwin-x64"}

      {"aarch64", "apple", "darwin"} ->
        {:ok, "darwin-arm64v8"}
    end
  end
97

I donā€™t have windows machine to test/debug, so NIF for windows is not supported yet. Even though Sharp/libvips can support windows, NIF does not. You can track progress here.

That said, if you are using wsl in windows, it should work.

@kwando can you share bit more details? which version of vix are you using? if possible please share compilation logs

98

Thank you! Iā€™ll follow up at github with any notes.

99

I have not tinkered that much with it, just tried to get image to run on the RPi4 :slight_smile:

āÆ mix compile
==> vix
make[1]: Entering directory '/home/kwando/vix_test/deps/vix/c_src'
** (CaseClauseError) no case clause matching: {"armv7l", "linux", "gnueabihf"}
    /home/kwando/vix_test/deps/vix/build_scripts/precompiler.exs:39: Vix.LibvipsPrecompiled.cast_target/1
    /home/kwando/vix_test/deps/vix/build_scripts/precompiler.exs:30: Vix.LibvipsPrecompiled.url/2
    /home/kwando/vix_test/deps/vix/build_scripts/precompiler.exs:16: Vix.LibvipsPrecompiled.fetch_libvips/0
    (elixir 1.12.3) lib/code.ex:1261: Code.require_file/2
make[1]: *** [Makefile:84: /home/kwando/vix_test/_build/dev/lib/vix/priv/precompiled_libvips] Error 1
make[1]: Leaving directory '/home/kwando/vix_test/deps/vix/c_src'
make: *** [Makefile:5: all] Error 2
could not compile dependency :vix, "mix compile" failed. You can recompile this dependency with "mix deps.compile vix", update it with "mix deps.update vix" or clean it with "mix deps.clean vix"
==> vix_test
** (Mix) Could not compile with "make" (exit status: 2).
You need to have gcc and make installed. If you are using
Ubuntu or any other Debian-based system, install the packages
"build-essential". Also install "erlang-dev" package if not
included in your Erlang/OTP version. If you're on Fedora, run
"dnf group install 'Development Tools'".

kwando@pi ļƒ§ 10.0.0.7 ~/vix_test took 4s 
āÆ mix hex.info
Hex:    2.0.6
Elixir: 1.12.3
OTP:    24.1.3

mix.lock

%{
  "castore": {:hex, :castore, "0.1.22", "4127549e411bedd012ca3a308dede574f43819fe9394254ca55ab4895abfa1a2", [:mix], [], "hexpm", "c17576df47eb5aa1ee40cc4134316a99f5cad3e215d5c77b8dd3cfef12a22cac"},
  "cc_precompiler": {:hex, :cc_precompiler, "0.1.7", "77de20ac77f0e53f20ca82c563520af0237c301a1ec3ab3bc598e8a96c7ee5d9", [:mix], [{:elixir_make, "~> 0.7.3", [hex: :elixir_make, repo: "hexpm", optional: false]}], "hexpm", "2768b28bf3c2b4f788c995576b39b8cb5d47eb788526d93bd52206c1d8bf4b75"},
  "elixir_make": {:hex, :elixir_make, "0.7.6", "67716309dc5d43e16b5abbd00c01b8df6a0c2ab54a8f595468035a50189f9169", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}], "hexpm", "5a0569756b0f7873a77687800c164cca6dfc03a09418e6fcf853d78991f49940"},
  "image": {:hex, :image, "0.27.0", "f1bc493fc2abe48a530292dbfc11bd2bbd00ab95289b46c47f096fafd3a4ec40", [:mix], [{:bumblebee, "~> 0.2", [hex: :bumblebee, repo: "hexpm", optional: true]}, {:evision, "~> 0.1.26", [hex: :evision, repo: "hexpm", optional: true]}, {:exla, "~> 0.5", [hex: :exla, repo: "hexpm", optional: true]}, {:jason, "~> 1.4", [hex: :jason, repo: "hexpm", optional: true]}, {:kino, "~> 0.7", [hex: :kino, repo: "hexpm", optional: true]}, {:nx, "~> 0.5", [hex: :nx, repo: "hexpm", optional: true]}, {:phoenix_html, "~> 2.14 or ~> 3.2", [hex: :phoenix_html, repo: "hexpm", optional: false]}, {:plug, "~> 1.13", [hex: :plug, repo: "hexpm", optional: true]}, {:sweet_xml, "~> 0.7", [hex: :sweet_xml, repo: "hexpm", optional: false]}, {:vix, "~> 0.15", [hex: :vix, repo: "hexpm", optional: false]}], "hexpm", "93d3383296dd2b1a0a18903b184e519e846e387943a83e3713b2b26ee7e3cec1"},
  "phoenix_html": {:hex, :phoenix_html, "3.3.1", "4788757e804a30baac6b3fc9695bf5562465dd3f1da8eb8460ad5b404d9a2178", [:mix], [{:plug, "~> 1.5", [hex: :plug, repo: "hexpm", optional: true]}], "hexpm", "bed1906edd4906a15fd7b412b85b05e521e1f67c9a85418c55999277e553d0d3"},
  "sweet_xml": {:hex, :sweet_xml, "0.7.3", "debb256781c75ff6a8c5cbf7981146312b66f044a2898f453709a53e5031b45b", [:mix], [], "hexpm", "e110c867a1b3fe74bfc7dd9893aa851f0eed5518d0d7cad76d7baafd30e4f5ba"},
  "vix": {:hex, :vix, "0.16.2", "1d18624c51a12ccd9042e263ba5b9145f4f6b06042d5f7fb170667f7e3bff42b", [:make, :mix], [{:castore, "~> 0.1", [hex: :castore, repo: "hexpm", optional: false]}, {:cc_precompiler, "~> 0.1.4 or ~> 0.2", [hex: :cc_precompiler, repo: "hexpm", optional: false]}, {:elixir_make, "~> 0.7.3 or ~> 0.8", [hex: :elixir_make, repo: "hexpm", optional: false]}, {:kino, "~> 0.7", [hex: :kino, repo: "hexpm", optional: true]}], "hexpm", "2c62dd442515641967e6286785785983256315e59aa1e8f29b3365d6854f7a11"},
}

The build-essential package is installed :confused:

āÆ which gcc
/usr/bin/gcc

āÆ which make
/usr/bin/make
100

Awesome work @kip! I forked your Livebook to download the exported yolo onnx file from GH for those that want to kick the tires without having python 3.10 installed:

Run in Livebook

5 Likes