Image - an image processing library based upon Vix

[akash-akya]

Thanks for sharing the details. Looks like you are running 32bit OS, so arch is getting reported as arm7l not aarch64. It’s works on my RPi4 which is running 64bit OS

pi@raspberrypi:~ $ lscpu
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
Vendor ID:                       ARM
Model:                           3
Model name:                      Cortex-A72
Stepping:                        r0p3
CPU max MHz:                     1800.0000
CPU min MHz:                     600.0000
BogoMIPS:                        108.00
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Vulnerable
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm crc32 cpuid

I’ll try to fix it release a version soon

[kip]

@chrismccord

download the exported yolo onnx file from GH

Great idea @chrismccord - I updated the repo Livebook to use the same approach. Much easier to experiment that way. Thanks!

[kip]

Image version 0.28.0 is just out. This version brought to you by @sodapopcan who worked out how to implement warp perspective using Nx and vix. Thanks for the PR and collaboration - that was a lot of fun. The changelog is:

Bug Fixes

• Fixes resizing images with alpha bands. Resizing images with alpha bands requires premultiplying the alpha band into the image and reversing that after resizing.

Enhancements

• Adds Image.warp_perspective/4 to apply a warp transformation to an image. Thanks very much to @sodapopcan for the PR and collaboration.

• Adds Image.straighten_perspective/3 to apply a warp perspective that straighens the image. It is basically Image.warp_perspective/4 with algorithmically derived destination parameter.

• Adds Image.map/2 which applies a transformation matrix (not a convolution kernel) to an image. A transformation matrix can be returned from Image.transform_matrix/3. These two functions underpin Image.warp_perspective/4.

• Adds Image.crop/2 which is a variation of Image.crop/5 that takes the same format of bounding box as that applied to Image.warp_perspective/4 making it easy to apply a warp transform and then crop to the transformed area. Note that cropping requires that the bounding box be a rectangle, not an arbitrary quadrilateral.

[kip]

Here’s an example of warp perspective (Livebook coming soon):

{:ok, image} = Image.open("./test/support/images/warp_perspective.jpg")

# Define the source coordinates (any quadrilateral)
iex> source = [{139, 125}, {826, 74}, {796, 559}, {155, 483}]
# And the destination coordinates (any quadrilateral, but often a rectangle)
iex> destination = [{139, 125}, {815, 125}, {815, 528}, {139, 528}]
# Warp the image
iex> {:ok, warped} = Image.warp_perspective(image, source, destination)

# Its common to crop to the subject of interest after warping
# so `Image.crop/2` can take the same `destination` data and
# crop to that area.
{:ok, cropped} = Image.crop(warped, destination)

[greven]

This library keeps getting better and better. Outstanding work. Thanks Kip et al.

[kwando]

Amazing Cant get it to run though it complains about my vectors and I’m stuck

EDIT:
I can get it to work with the test/support/validate/warp_perspective.jpg file you have.I do not understand what that error tries to tell me in my case.

[sodapopcan]

Also thank you to @akash-akya and @jerdew for refactoring and advice that wasn’t reflected in the git history!

[kip]

@kwando, please send me a DM with the image or a link to it if you wouldn’t mind? Even better, open an issue on https://github/elixir-image/image/issues so I can bottom this out and not pollute the forum. I’ll jump on this right away.

[kwando]

I got the new warp_perspective to work And I managed to write a script that swaps a billboard/sign

[warmwaffles]

It’s not entirely clear to me looking at libvips or Image on how I could resize an image and not keep the aspect ratio. Can anyone point me in the right direction?

[sodapopcan]

thumbnail is likely what you want.

[kwando]

Yeah, thumbnail would work.

Image.thumbnail!(my_image, "400x400", resize: :force)

[tmariaz]

FYR…
I recently encountered prediction problem. When trying to predict some urls, it doesn’t return anything. Upon consulting with @kip it has to do with top_k: 1. But the latest version image version 0.28.2 fixed the issue.

I’m just sharing my experience here. So if anyone came across the issue then should know what to do as well.

This code wasn’t working

    image_data = %HTTPoison.Response{body: body} = HTTPoison.get!(image_url)
    my_image = image_data.body
    classify(my_image)

@kip suggestion:

%HTTPoison.Response{body: image_data} = HTTPoison.get!(image_url)
{:ok, image} = Image.from_binary(image_data)
predictions = Image.Classification.classify(image)

Which basically fixed my problem.

iex> Image.Classification.classify(image)                                                                        
%{
  predictions: [
    %{label: "toyshop", score: 0.3621242344379425},
    %{label: "shopping basket", score: 0.1781061440706253},
    %{label: "hamper", score: 0.07665970921516418},
    %{label: "rubber eraser, rubber, pencil eraser", score: 0.06588040292263031},
    %{label: "carton", score: 0.04808710142970085}
  ]
}

# To get only the labels from the prediction
iex> Image.Classification.labels(image, min_score: 0.01)
["toyshop", "shopping basket", "hamper", "rubber eraser", "rubber",
 "pencil eraser", "carton"]
iex> Image.Classification.labels(image, min_score: 0.05)
["toyshop", "shopping basket", "hamper", "rubber eraser", "rubber",
 "pencil eraser"]
iex> Image.Classification.labels(image, min_score: 0.1) 
["toyshop", "shopping basket"]

Thanks to @kip for the help.

[kevinlang]

Great library!

next up in the list is another library I’m calling Imagine that is basically a plug to serve images using Image that can be very easily added to any Plug or Phoenix application. A sort of a mix of thumbor and Cloudinary. But dead simple to add to an Elixir application and really easy to use.

I was just thinking of this earlier today and came to this thread to mention the same thing . Looks like it is already in consideration, great to hear! I can see this being a boon to Phoenix applications.

Edit: even without client hints (which are still not supported by Firefox or Safari) such a library would still be quite useful. E.g., for my use case I’d use it to transform pngs I serve out of priv/static to webp on the fly.

[kip]

@kevinlang thanks for the encouragement. I’m working on the design in my head while I finish up to last ex_cldr_* libs. Then I want to get this done because I will use it for my new (to be built) web site.

Do you have a preference and views for any particular URL format? I’m going to make that pluggable. But I need to start somewhere and being compatible with existing services as far as possible is a goal.

[kevinlang]

I have no preference. My use cases would always be pretty simple of just a crop and a format change which would be similar enough across any URL format.

It may be worth considering doing something thumbor compatible, as it seems to the most established URL format for this sort of thing. A lot of “client” libraries across languages Libraries — Thumbor 7.2.0 documentation . Even Elixir has one.

It would be cool if the “client library” (the URL generator and signer) uses more or less the same piping style that Image itself uses.

However, I do not have much experience in this field. It may be that there exists better URL formats that are more flexible or easier to parse or whatever.

[kip]

Thanks, my default thinking was the Thumbor URL format first but I’m open to other opinions.

I am also thinking about the client library side. Compatibility with existing client libs is a good idea. I’ll check out the Elixir one and perhaps the author would be interested to collaborate.

I also have in mind a <.picture> Liveview component as one part of optimising the client experience.

[sodapopcan]

I totally missed the whole thing about Imagine.

Are you ahem imagining something that would allow the full use of Image through URLs or just basic stuff like resize?

[greven]

I’m building my website now, in full LiveView (because it has several dynamic / interactive things) and this is something I wanted to do.
Imagine is a great name by the way.

To be honest this could even be something you could monetise with a Pro version or something.

[kip]

The aim is pretty much everything Image can do. Using the URL structure of thumbor and/or Cloudinary. And for the Elixir client lib, basically the same as the current API of Image, but it builds a collection of transforms that a tree then converted to a URL.