Bumblebee is out: GPT2, Stable Diffusion, and More in Elixir (and LiveBook)

[pestoAioli]

Thank you Jose!

[kwando]

Thanks for the update Jose

I did what I could

[derpycoder]

I was figuring out how to run existing models. Because I am no expert in Machine Learning.

I looked at how others are doing it, Tensorflow Serving, etc. I wished Elixir had something like that.

Lo and behold: Bumblebee.

Machine learning serving from the comfort of Elixir.

How are people so awesome?!?

Thank you @josevalim.

[kip]

No surprise, image classification would be a great addition to image but for some reason I can’t fathom, starting up Nx.Serving is failing with:

06:30:13.408 [notice] Application image exited: exited in: Image.Application.start(:normal, [])
    ** (EXIT) exited in: GenServer.call(EXLA.Client, {:client, :host, [platform: :host]}, :infinity)
        ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started

Which suggests there is a failure starting EXLA.Client.

If I don’t configure EXLA then the app starts fine. But no surprise - the binary backend isn’t suited to this kind of work and I lose patience after 5 minutes of a test run

Any help would be appreciated - I’m pretty sure this is a stupid user error. Here’s the relevant info: (Mac ARM, OTP 25, Elixir 1.14)

# Image.Application
defmodule Image.Application do
  @moduledoc false
  use Application

  def start(_type, _args) do
    Supervisor.start_link(
      [
        {Nx.Serving, serving: Image.Classification.serving(), name: Image.Serving, batch_timeout: 100}
      ],
      strategy: :one_for_one
    )
  end
end

# Image.Classification
defmodule Image.Classification do
  alias Vix.Vips.Image, as: Vimage

  def serving(model \\ "microsoft/resnet-50", featurizer \\ "microsoft/resnet-50") do
    {:ok, model_info} = Bumblebee.load_model({:hf, model})
    {:ok, featurizer} = Bumblebee.load_featurizer({:hf, featurizer})

    Bumblebee.Vision.image_classification(model_info, featurizer,
      top_k: 1,
      compile: [batch_size: 10],
      defn_options: [compiler: EXLA]
    )
  end

  def classify(%Vimage{} = image) do
    with {:ok, binary} <- Image.to_nx(image) do
      Nx.Serving.batched_run(Image.Serving, binary)
    end
  end
end

# mix.exs
  defp deps do
    [
       ...
      # For NX interchange and
      # Bumblebee for image classification
      if(otp_release() >= 24, do: [
        {:nx, "~> 0.4.1", optional: true},
        {:bumblebee, "~> 0.1.0", optional: true},
        {:exla, "~> 0.4.1", optional: true}
      ]),
      ...
     ]
     |> List.flatten()
     |> Enum.reject(&is_nil/1)
  end

# dev.exs
config :nx,
  default_backend: EXLA.Backend

[seanmor5]

It looks like the EXLA client process died, usually this happens when startup fails like when there is a linking error or something in the NIF. Is there any additional error messages before you hit this point?

[kip]

@seanmor5 thanks for the assist. No other errors. I tried starting it manually from iex and perhaps the following helps a little?

Interactive Elixir (1.14.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Supervisor.start_link([{Nx.Serving, serving: Image.Classification.serving(), name: Image.Serving, batch_timeout: 100}])
** (exit) exited in: GenServer.call(EXLA.Client, {:client, :host, [platform: :host]}, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir 1.14.0) lib/gen_server.ex:1027: GenServer.call/3
    (exla 0.4.1) lib/exla/backend.ex:147: EXLA.Backend.client_and_device_id/1
    (exla 0.4.1) lib/exla/backend.ex:44: EXLA.Backend.from_binary/3
    (bumblebee 0.1.0) lib/bumblebee/conversion/pytorch/loader.ex:71: Bumblebee.Conversion.PyTorch.Loader.object_resolver/1
    (unpickler 0.1.0) lib/unpickler.ex:828: Unpickler.resolve_object/2
    (unpickler 0.1.0) lib/unpickler.ex:818: anonymous fn/2 in Unpickler.finalize_stack_items/2
    (elixir 1.14.0) lib/map.ex:924: Map.get_and_update/3
    iex:1: (file)

[kip]

Just to try to dive one level further I manually started EXLA.Client without error but of course then Nx.Serving won’t start:

iex(2)> Supervisor.start_link([EXLA.Client], strategy: :one_for_one)
{:ok, #PID<0.18669.0>}
iex(3)> Supervisor.start_link([{Nx.Serving, serving: Image.Classification.serving(), name: Image.Serving, batch_timeout: 100}])
2022-12-10 12:58:37.980550: I tensorflow/compiler/xla/pjrt/tfrt_cpu_pjrt_client.cc:214] TfrtCpuClient created.
                                                                                                              ** (exit) exited in: GenServer.call(EXLA.Defn.LockedCache, {:lock, {#Function<88.82845959/1 in EXLA.Backend.reshape/2>, [{{:f, 32}, {2048000}, [nil]}]}}, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir 1.14.0) lib/gen_server.ex:1027: GenServer.call/3

[seanmor5]

Oh I didn’t realize you were starting it up in this way. EXLA spins up a few processes: nx/application.ex at main · elixir-nx/nx · GitHub

That you will need to start to work with it

[seanmor5]

Though all of these should be started when you start the serving based on your code

[kip]

That was just a test to see if EXLA.Client would start. My configuration is, as best I can tell, exactly the same as that in the demo Phoenix app. It seems EXLA.Client can start (just as a test). But when starting Nx.Servering as a supervised child, the error in the original message occurs. I’m definitely stuck.

Just to make sure I force-compiled exla again and no errors:

kip@Kips-MacBook-Pro image % mix deps.compile exla --force
==> exla
Unpacking /Users/kip/Library/Caches/xla/0.4.1/cache/download/xla_extension-aarch64-darwin-cpu.tar.gz into /Users/kip/Development/image/deps/exla/cache
c++ -fPIC -I/Users/kip/.asdf/installs/erlang/25.1/erts-13.1/include -Icache/xla_extension/include -O3 -Wall -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -shared -std=c++17 -w -DLLVM_ON_UNIX=1 c_src/exla/exla.cc c_src/exla/exla_nif_util.cc c_src/exla/exla_client.cc -o cache/libexla.so -Lcache/xla_extension/lib -lxla_extension -flat_namespace -undefined suppress
install_name_tool -change bazel-out/darwin_arm64-opt/bin/tensorflow/compiler/xla/extension/libxla_extension.so @loader_path/xla_extension/lib/libxla_extension.so -change bazel-out/darwin-opt/bin/tensorflow/compiler/xla/extension/libxla_extension.so @loader_path/xla_extension/lib/libxla_extension.so cache/libexla.so
Compiling 21 files (.ex)
Generated exla app

[wojtekmach]

I noticed you have exla as an optional dep. I don’t remember off the top of my head but optionals may not be automatically started. Try Application.ensure_all_started(:exla) before the call site.

[kip]

@wojtekmach thank you so much. As I suspected, totally a user error and a pretty silly one! Starts up perfectly fine now. Sorry for the noise. Very excited by what this represents for several of my libraries!

Example

iex> Image.open!("./test/support/images/lamborghini-forsennato-concept.jpg") 
...> |> Image.Classification.classify()
%{predictions: [%{label: "sports car, sport car", score: 0.9962735176086426}]}

[bdarla]

BTW, just recently I found the following article that combines Phoenix, Axon and Elasticsearch.

It focuses on semantic search on text and not on images. I have not reproduced its content yet. Just note that it also involves some Python.

[seanmor5]

@bdarla I’m going to update that post soon to be 100% Elixir with Axon, Bumblebee, and a library that completely eliminates the need for an additional service in Elasticsearch

[Pistrie]

[olivermt]

I would have to check but I dont think we sell any nipples

[kanonk]

Amazing work!

To check if GPU is supported, visit GitHub - elixir-nx/xla: Pre-compiled XLA extension and see usage of XLA_TARGET. To load the correct version, set the environment variable XLA_TARGET to the desired version (I used XLA_TARGET=cuda118 for an RTX 3090 card). Make sure that the version you choose is installed on your system (e.g. CUDA).

[seanmor5]

FYI, the update post for this will be following this one: https://dockyard.com/blog/2023/01/04/search-and-clustering-with-exfaiss