marinakr

marinakr

Troubleshooting: Bumblebee audio transcription provides same text for few parsed webm files

Hi, I am trying to use Bumblebee to transcript audio to text
I have few multiple *.webm files with audio that sounds like ‘text audio with Bumblebee’, ‘one, two, three, four, five’, etc
I can reproduce it with mediaplayer and all file contains different audio
However, looks like bumblebee always generates same prediction with text ’ you’:

predicitons: %{
  chunks: [
    %{text: " you", start_timestamp_seconds: nil, end_timestamp_seconds: nil}
  ]
}
# ..x5+ times

predicitons: %{
  chunks: [
    %{      text: " Thank you.", start_timestamp_seconds: nil, end_timestamp_seconds: nil    }
  ]
}
predicitons: %{
  chunks: [
    %{text: " Bye.", start_timestamp_seconds: nil, end_timestamp_seconds: nil}
  ]
}

(predicitons is IO.inspect label in here)

My code:
Application application.ex, child spec:

children = [
      ........
      {Nx.Serving,
       serving: create_audio_serving(),
       name: Recognizer.AudioServing,
       batch_size: 4,
       batch_timeout: 100}
       .....
    ]

defp create_audio_serving() do
    # Load the pre-trained model
    {:ok, model_info} = Bumblebee.load_model({:hf, "openai/whisper-tiny"})
    {:ok, featurizer} = Bumblebee.load_featurizer({:hf, "openai/whisper-tiny"})
    {:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "openai/whisper-tiny"})
    {:ok, generation_config} = Bumblebee.load_generation_config({:hf, "openai/whisper-tiny"})

    Bumblebee.Audio.speech_to_text_whisper(model_info, featurizer, tokenizer, generation_config,
      compile: [batch_size: 4],
      defn_options: [
        compiler: EXLA
      ]
    )
  end

In worker process I have code:

defmodule Recognizer.Room do
  @moduledoc false

  use GenServer, restart: :temporary
  ...
  @impl true
  def handle_cast({:receive_audio_msg, audio_base64}, state) do
    file = "/tmp/audio-#{state.id}.webm"
    audio_data = Base.decode64!(audio_base64)

    File.write!(file, audio_data)
    reader = Xav.Reader.new!(file, read: :audio)

    case Xav.Reader.next_frame(reader) do
      {:ok, frame} ->
        tensor = Xav.Frame.to_nx(frame)
        Task.async(fn -> Nx.Serving.batched_run(Recognizer.AudioServing, tensor) end)

      {:error, :no_keyframe} ->
        Logger.warning("Couldn't decode audio frame - missing keyframe!")
    end

    {:noreply, state}
  end

@impl true
  def handle_info({_ref, predicitons}, state) do
    predicitons |> IO.inspect(label: :predicitons) 
   {:noreply, state}
  end

I have added default exla backend like it is suggested in very similar question but predicition still nonsense
What I am doing wrong and why I have this random predictions?

Marked As Solved

marinakr

marinakr

The issue with Xav.next_frame
Probably it takes only first frame
Passing entire file like it was done in cool-whisper-server works

iex(10)> Nx.Serving.batched_run(Recognizer.AudioServing, {:file, "/home/maryna/Music/audio-17405164795554743983100781892.webm"})
%{
  chunks: [
    %{
      text: " Finally in temporary digital storage.",
      start_timestamp_seconds: nil,
      end_timestamp_seconds: nil
    }
  ]
}

Also Liked

lawik

lawik

Nerves Core Team

How much data is in a frame? Collect a bunch of them. Stick em together, then pass them through.

I have done this in here. Checking if the concatenated bytes are above a desired size.

Something like this:
https://github.com/lawik/membrane_transcription/blob/9b2519d85ed93ba9dac46fb6ba7b105533b74160/lib/membrane_transcription/element.ex#L93

Old version of Membrane and all that but I use similar mechanisms in this one which is more up to date.

Last Post!

lawik

lawik

Nerves Core Team

How much data is in a frame? Collect a bunch of them. Stick em together, then pass them through.

I have done this in here. Checking if the concatenated bytes are above a desired size.

Something like this:
https://github.com/lawik/membrane_transcription/blob/9b2519d85ed93ba9dac46fb6ba7b105533b74160/lib/membrane_transcription/element.ex#L93

Old version of Membrane and all that but I use similar mechanisms in this one which is more up to date.

Where Next?

Popular in Questions Top

minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

Other popular topics Top

minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 31494 112
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 40042 209
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New

We're in Beta

About us Mission Statement