Nx and Whisper, "cannot build an empty tensor"

Elixir Framework Forums Nx Forum
,
1

Using the live_beats transcoding as a basis for this code, I’m transcribing an audio file. Sometimes, though, the very last segment will fail the entire batch run because it will attempt to transcribe silence.

cannot build an empty tensor

[error] Task #PID<0.2261.0> started from #PID<0.1557.0> terminating
** (ArgumentError) cannot build an empty tensor
    (nx 0.6.4) lib/nx.ex:2062: Nx.from_binary/3
    (cool_app 1.5.1) lib/cool_app/multimedia/audio.ex:10: anonymous fn/3 in CoolApp.Audio.speech_to_text/3
    (elixir 1.14.2) lib/task/supervised.ex:89: Task.Supervised.invoke_mfa/2
    (elixir 1.14.2) lib/task/supervised.ex:34: Task.Supervised.reply/4
    (stdlib 4.2) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Function: &:erlang.apply/2
    Args: [#Function<0.33943372/1 in CoolApp.Audio.speech_to_text/3>, [3740]]

I’d like to catch the error and ignore it, possbily in the Enum.map. I thought of a case do but I want to avoid generating a nil and having that returned.

Suggestions on how to “escape hatch” out of this and avoid losing the batch job?

cribbed from live_beats:

  def speech_to_text(path, chunk_time, func) do
    {:ok, stat} = CoolApp.MP3Stat.parse(path)

    0..stat.duration//chunk_time
    |> Task.async_stream(
      fn ss ->
        args = ~w(-ac 1 -ar 16k -f f32le -ss #{ss} -t #{chunk_time} -v quiet -)
        {data, 0} = System.cmd("ffmpeg", ["-i", path] ++ args)
        {ss, Nx.Serving.batched_run(WhisperServing, Nx.from_binary(data, :f32))}
      end,
      max_concurrency: 2,
      timeout: :infinity
    )
    |> Enum.map(fn 
      {:ok, {ss, %{results: [%{text: text}]}}} -> func.(ss, text) 
    end)
  end
2

I made a workaround which seems to fix the problem when an empty tensor is generated(it happens when the audio ends precisely at a moment divisible by 20 seconds).

      0..stat.duration//chunk_time
      |> dbg()
      |> Task.async_stream(
        fn ss ->
          args = ~w(-ac 1 -ar 16k -f f32le -ss #{ss} -t #{chunk_time} -v quiet -)
          {data, 0} = System.cmd("ffmpeg", ["-i", path] ++ args)
+          if byte_size(data) > 0 do
+            {ss, Nx.Serving.batched_run(WhisperServing, Nx.from_binary(data, :f32))}
+          else
+            {ss, ""}
+          end
        end,
        max_concurrency: 2,
        timeout: :infinity
      )
      |> Enum.map(fn
+        {:ok, {ss, %{results: [%{text: text}]}}} ->
+          func.(ss, text)
+        {:ok, {ss, ""}} ->
+          func.(ss, "")
      end)