Image - an image processing library based upon Vix

Your Libraries & OS Mentoring Libraries
41

I have some ideas I would like to share with you, since I can’t do them :sweat_smile::

  1. Color palette detection, for theming purposes. (Like say a page displays an image, it can color itself based on the palette of the image, or it can be used for showing a dominant color, before the image loads.)
  2. Color grading images based on LUT provided. (For consistent look and feel of the site)
  3. Configurable QR code. (Like QR code monkey)
  4. Rendering a HEEX template, a webpage or a section of a webpage, or svg into an image without using headless chrome if possible. (To allow creation of badges like shields.io, or run in LiveBook badge, or images that can be used in profiles like GitHub or Twitter or Linked, like GitHub Readme Stats)
  5. Video to gif or something better.
  6. Some default / custom placeholder image in case of error.

Documentation, Tutorial, interactive LiveBooks, Cheatsheet:

  1. For instance, how do I introduce a ML trained model to the pipeline. (E.g. Object detection using YOLO ML Algo, or Image scale using a machine model.)
  2. How to introduce compression step into the pipeline.
  3. General examples like the ones you have put in this discussion thread are missing in the official docs page. (I would have not been able to imagine such things are possible by looking at the API :sweat_smile:)

P.S. I skimmed through the API Documentation, so I don’t know if some of the suggestions are already possible. I hope there are examples in the future for every thing this library is capable of.

P.P.S. It’s incredible how many APIs are present in that library, so I feel ashamed asking for more. :sweat_smile:

1 Like
42

I appreciate the suggestions, they are very welcome. Some may be possible.

  1. Color palette detection, for theming purposes. (Like say a page displays an image, it can color itself based on the palette of the image, or it can be used for showing a dominant color, before the image loads.)

I’ve got another lib called Color in development that implements color science and John Cullpit, the author of libvips, has agreed to put k-means of an image in the next release which is very helpful for palette optimisation.

  1. Color grading images based on LUT provided. (For consistent look and feel of the site)

Color grading I also want to do since photography is a passion of mine and managing color is very helpful. May be possible, there are some LUT capabilities in libvips and vision. Also complementary, analogous, split complementary, triadic color calculation.

  1. Configurable QR code. (Like QR code monkey)

Nothing special in libvips for that but I’ll take a look. Lower priority in my mind.

  1. Rendering a HEEX template, a webpage or a section of a webpage, or svg into an image

Not available directly in libvips. It renders SVG into images really well using librsvg but not html.

  1. Video to gif or something better.

I’m more likely to do gif to video since that the more common trend I’m seeing?

  1. Some default / custom placeholder image in case of error.

Yes, that makes sense. Will think about what that might look like.

  1. Documentation

Absolutely - I will work on tutorial and user guide style docs using Livebook as the platform as a priority before version 1.0. It all takes a lot of time …

3 Likes
43

Amazing work on this lib kip, can see this becoming the defacto lib for Elixir regarding common Image tasks.
The idea is not to wrap OpenCV (eVision) completely right, just the most common image tasks with higher level abstractions on top, I guess?

Regarding the suggestion of derpycoder, I have been actually thinking in building and OG Image generator (that would plug into… plug / Phoenix) inspired by recent release of NextJS OG Image: Introducing OG Image Generation: Fast, dynamic social card images at the Edge – Vercel.

I guess this makes more sense as a separate library, I think? The idea is to leverage librsvg as well.

1 Like
44

Yes, you are right. Both Vix and eVision are great libraries and there’s little value in just wrapping them. But there is value, I think, providing more idiomatic interfaces to common higher level image processing functions.

I also hope that this work demonstrates that image processing on Elixir is definitely possible - and not just possible, really can be a joy leveraging everything Elixir brings to the table as well as the power of the underlying libs.

One of the unifying reasons why Image builds on top of both Vix and eVision is the ability for both libraries to reuse binary image data without copying it. That makes interoperation really performant. For example, for QR code generation the matrix is generated in eVision but resized using Vix. But no data is copied. Similar for the Nx integration.

It probably doesn’t make sense to wrap any other libraries if they don’t also have a compatible in-memory model. However given the explosion of imaging capabilities I would like to find a way to keep the imaging eco-system as easy to adopt at the higher level as possible without losing the ability to go as low as you want.

As for OG Image generation, if the rendering can be accomplished by librsvg then Vix and/or Image might be a fast way to get started. I didn’t look closely enough but it did look somewhat SVG-centric. I just haven’t explored if librsvg supported HTML embedded SVG. And I don’t think it will support tailwind classes out of the box (I see the Vercel lib uses the same class names but with a different attribute name?).

Overall, just happy to see an emerging vibrant community of people interested in image processing and some amazing tooling being build, especially by @akash-akya and @cocoa.

2 Likes
45

I wanted to tell you about blurhash, for placeholder purposes, which I forgot to link in my previous comment!!

They have a funny description too:

Does your designer cry every time you load their beautifully designed screen, and it is full of empty boxes because all the images have not loaded yet? Does your database engineer cry when you want to solve this by trying to cram little thumbnail images into your data to show as placeholders?

BlurHash generates incredibly small previews using Base83 encoding. Base83 encoding allows it to squeeze more information into fewer bytes, which is part of how it keeps the previews so small.

{
  "blurhash" : "UAA]{ox^0eRiO_bJjdn~9#M_=|oLIUnzxtNG",
  "w" : 276,
  "h" : 400
}

Which means in front end JavaScript is needed for preview!

An alternative to generate blurry placeholder, that doesn’t need JavaScript on front end is SQIP, which is based on SVG:

A great side-effect of using SVG as the basis for image previews is that SVG is a plaintext asset and therefore benefits from compression by gzip, zopfli and brotli. Using these compression techniques cuts SQIP size down by half, reaching the minimum viable raster LQIP size of ~400-600 bytes while providing a superior user experience.

image
image1200×2340 216 KB

1 Like
46

I already done such thing in the past, but I was using ImageMagick via shell. With Image that would be much easier nowadays.

47

Thats very interesting, thanks for posting. I especially like that the original primitive is based upon libvips so I think there’s a probably a reasonable path forward. And I like the idea of SVG-based placeholders since they inline well and compress well.

Also, it’s just a really interesting algorithm and artistic outputs.

Hmmm, looks like there’s another rabbit hole I’m diving down … :slight_smile:

4 Likes
48

I’m definitely interested in what primitive does and will investigate doing that with eVision/OpenCV.

Of the other processes indicated on the SQIP page:

  1. Blurring is already implemented as Image.blur/2
  2. Pixelation is now on the main branch as Image.pixelate/2
  3. Posterisation will come when libvips 8.14 lands with a public api to libimagequant

Combined with the other image optimisations available on save I think there are some good options today - with some better options to come.

2 Likes
49

Is it possible to compress the Animated PNGs with Image library?

I tried several programs, and even libvips, but couldn’t make it work!

vipsthumbnail beaming_face.png[n=-1] -o beaming.png -s 64
vips resize beaming_face.gif[n=-1] -o beaming.gif -s 0.25

I get: no matches found: beaming_face.gif[n=-1]

beaming_face

I want to use Microsoft Fluent Emojis, I couldn’t find their animated SVG counterpart.

P.S. After a lot of searching, I found out how to resize using ffmpeg:

For APNG:
ffmpeg -i input.png -vf scale=64x64 -f apng output-64x64.png
ffmpeg -I input.png -vf scale=32x32 -f aping output-32x32.png

For GIF: (Pasted here for demo, since Discourse didn’t support APNG upload!)
ffmpeg -i beaming_face.png -filter_complex "[0:v] scale=100:100, split [a][b]; [a] palettegen=reserve_transparent=on:transparency_color=ffffff [p]; [b][p] paletteuse" output-100x100.gif
output-100x100

Hello, Reddit style animated emojis!!

P.P.S. I still hope that Image library has a easy method, so I can resize a whole folder worth of emojis.

50

It would be great if Image.Options.Crop.crop_focus() could have a focal point {x_percent, y_percent} too. It’s really nice to have an Image in your backend with a little dot you can drag to choose the focal point of the image. Then we can crop around this point:

image
image1068×766 38.1 KB

Here it’s x=17 and y=48 (in percent) which means if I had a portrait crop here it would crop around this point (as far as is possible of course). It’s pretty neat!

51

Hey I just open-sourced a library that we use in production that do just that (with the imaginary format, a popular go image proxy server). We could abstract the parsing of the URL to be compatible with other popular tools. Right now it supports streaming images from arbitray URL, url signature of useful transform operation from Vix.

2 Likes
52

It would be great if Image.Options.Crop.crop_focus() could have a focal point {x_percent, y_percent}

Great suggestion and happy to do that. Will get that out the door this weekend.

The process will be a little more expensive since Image.thumbnail/3 doesn’t take that option natively. So basically I will crop from the provided centre-point and then set the :centre option to :focus to get the result you’re after.

3 Likes
53

That looks great! Really happy to see more libraries building on Vix.

54

I think there is an issue with the typespec here. When calling

Image.write!(vips_image, :memory, suffix: ".png")

a binary is returned, the spec states than an %Vix.Vips.Image is returned.

Maybe I’m using it wrong, but this works (but dialyzer unhappy).

vips_image |> Image.write!(:memory, suffix: ".png") |> Base.encode64()
1 Like
55

Fixed on main branch if you’re willing to try it from there. I have a couple of additional tests and then will publish an update to hex. Thanks for pointing out the issue.

4 Likes
56

You missed the spec for Image.write! which still states that Vimage.t() | no_return() is returned. I changed the code to use write() and dialyzer is happy. (As am I, very nice lib!)

1 Like
57

Arrrrghhhhhhh thanks again @Sebb, on it!

2 Likes
58

@kip

I know I’m asking for the moon and the stars :slight_smile: Is there a way to put text on an image within a bounding box?

Let’s say, I need to add a caption on top of an image in the lower half of it, but I don’t know beforehand how long the text is? Basically, best fit caption from ImageMagick?

59

Definitely possible to do, will need a few hours to get back with the details (day job getting in the way). Might be a bit tricky to auto-size the text to make it fit in a bounding box. But putting text in a bounding box - can do.

3 Likes
60

libvips does best-fit by default. For example, using vips CLI

$ vips text out.png "This text is resized to best fill the space given." --width 320 --height 140 --rgba

out

I think you just have to map width and height options to Image lib.

3 Likes