Transform Nx tensor image to web image

Elixir Framework Forums Nx Forum
1

Hello everyone,

I am testing stable diffusion with Bumblebee. Everything is fine, but I would like to display the generated images on the web.

I receive these kind of results…

%{
  results: [
    %{
      image: #Nx.Tensor<
        u8[512][512][3]
        EXLA.Backend<host:0, 0.3536702673.2258239536.2772>
        [
          [
            [75, 103, 11],
            [68, 107, 0],
            [64, 109, 0],
            [63, 110, 0],
            [62, 107, 0],
            [61, 110, 0],
            [61, 110, 0],
            [60, 110, 0],
            [60, 110, 0],
            [60, 110, 0],
            [61, 109, 0],
            [61, 109, 0],
            [61, 109, 0],
            [60, 108, 0],
            [60, 109, 0],
            [60, 109, ...],
            ...
          ],
          ...
        ]
      >,
      is_safe: true
    },
...

It is a list of #Nx.Tensor<u8[512][512]…>

In the demo code these images are transformed into Kino.Image, to be displayed into Livebook.

Does anyone knows how to transform those images into something more usual for the web, like jpg, or png?

Thanks for taking time

1 Like
2

Sorry for the noise…

I added Image, and used the from_nx to transform resulting image to Vix.Vips.Image

From there it’s possible to dump a jpg.

Here is the sample with the default parameters (“numbat, forest, high quality, detailed, digital art”, “darkness, rainy, foggy”)

test

Have a nice weekend

4 Likes
3

That’s really cool Koko!

Do you know if it would be easy enough to feed it a photo and do something like ‘add a Christmas hat to the person or people in this photo’? I’m sure some of the people on one of the sites I admin would love that :lol:

The whole image manipulation side of Nx/Bumblebee is really exciting, it’s re-igniting my passion for a couple of projects that I had in mind!

4

I guess it should be possible to use previous pictures, but I did not try it.

1 Like
5

I wonder if @kip knows (or whether that’s part of your plans for Image, Kip?) I think it would be a killer feature and I can think of loads of use-cases :smiley:

6

I’ve just now finished up the first integration between Image and Bumblebee which is image classification. I haven’t looked at StableDiffusion integration - will take a look and see what that might look like (but maybe not til early new year unless someone would like to dive in and contribute). I’m also looking forward to the work @seanmor5 is considering for object detection and image segmentation.

3 Likes
7

I ended up using stb_image for that, then it gets pretty straighforward:

  @spec tensor_image_to_base64(Nx.Tensor.t()) :: String.t()
  defp tensor_image_to_base64(tensor) when is_struct(tensor, Nx.Tensor) do
    "data:image/png;base64,#{tensor |> StbImage.from_nx() |> StbImage.to_binary(:png) |> Base.encode64()}"
  end

Bumblebee is really cool though, I got to play with it and was really impressed, here’s the result sample app with few models: GitHub - bcat-eu/jimmy: Your friendly robot

4 Likes
8

Nice, I didn’t knew about stb_image. And by the same person who did evision :slight_smile:

2 Likes
9

Well, I didn’t know about evision, looks like another interesting lib in that space :slight_smile:

2 Likes