kokolegorille

Speech to text - are there are tools that can convert an audio source to text?

Hello everyone,

I would like to search/index videos by keywords.

The first step to do this is to translate audio to text. I know of some clouds providers that does speech-to-text, but before this, I would like to understand what I would need to roll my own.

I know the opposite, text-to-speech, is quite easy with some open source tools.

Did You meet similar task of converting audio source to text? If so do You know open source libraries that could help me achieve this?

Thanks for taking time

21 comments

#elixir #speech-to-text

37 4764 21

2023-01-24 05:18:12 UTC

Marked As Solved

idi527

GitHub - mozilla/DeepSpeech: DeepSpeech is an open source embedded (offline, on-device) speech-to-text engine which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers. · GitHub is rather easy to use, but it’s very slow, at least on CPU. To transcribe a two second audio clip of me saying “hello, hello” it takes 10 seconds. But it’s quite accurate.

I also had to use ffmpeg to transcode the recording into a format deepspeech understands:

ffmpeg -i New\ Recording\ 3.m4a -acodec pcm_s16le -ac 1 -ar 16000 audio.wav

Post #2

Also Liked

kokolegorille

For those interested in the subject, I did some basic tests with GitHub - mozilla/DeepSpeech: DeepSpeech is an open source embedded (offline, on-device) speech-to-text engine which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers. · GitHub

$ deepspeech --model deepspeech-0.5.1-models/output_graph.pbmm --alphabet deepspeech-0.5.1-models/alphabet.txt --lm deepspeech-0.5.1-models/lm.binary --trie deepspeech-0.5.1-models/trie --audio audio/8455-210777-0068.wav 
Loading model from file deepspeech-0.5.1-models/output_graph.pbmm
TensorFlow: v1.13.1-10-g3e0cc5374d
DeepSpeech: v0.5.1-0-g4b29b78
2019-09-27 01:53:25.693154: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "UnwrapDatasetVariant" device_type: "GPU" host_memory_arg: "input_handle" host_memory_arg: "output_handle"') for unknown op: UnwrapDatasetVariant
2019-09-27 01:53:25.693187: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "UnwrapDatasetVariant" device_type: "CPU"') for unknown op: UnwrapDatasetVariant
2019-09-27 01:53:25.693200: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "WrapDatasetVariant" device_type: "GPU" host_memory_arg: "input_handle" host_memory_arg: "output_handle"') for unknown op: WrapDatasetVariant
2019-09-27 01:53:25.693211: E tensorflow/core/framework/op_kernel.cc:1325] OpKernel ('op: "WrapDatasetVariant" device_type: "CPU"') for unknown op: WrapDatasetVariant
Loaded model in 0.0174s.
Loading language model from files deepspeech-0.5.1-models/lm.binary deepspeech-0.5.1-models/trie
Loaded language model in 1.38s.
Running inference.
your power is sufficient i said
Inference took 1.562s for 2.590s audio file.

It looks promising and is even faster than expected. And with a compatible card, it is possible to run on GPU

Thanks again @idi527 for the link. Now I need to glue ffmpeg and deepspeech with Elixir to have the beginning of a working solution.

Post #4

marcelfahle

Hello Christian, I’m curious about your findings as well. My Gist so far:

Whisper is great at speech recognition, has timing issues though and I’m not yet sure how to deal with silences in speech (I want to hide subs for example when nobody is talking.. use VAD Voice Activity Detection on top? Dunno ) I ran my own and also on replicate.com, which is pretty fast (1h file in 90 secs)
Sonix feels like the goat to me. Similarly good in regards of speech as whisper, but way better timings, speaker diarization, etc, etc. BUT: expensive!
rev.ai - is what I’m currently using for my Video CMS. It’s actually pretty good and timings are very accurate. It’s much cheaper than Sonix and I feel gives me the best bang for the buck right now (I’m currently torn between sonix and rev )
aws transcribe - feels “ok”. timings are good, speech in my examples maybe a little less accurate than rev.ai
fireflies - Cool tool, didn’t play with it that much yet. I think they might use whisper under the hood, as I see a bunch of weird words that whisper was able to catch successfully (none of the others did), and they caught them too. They also have timing issues.
deepgram - feels very similar what aws and rev give me on my first few tries. pretty cheap though and lots of features, so I will play some more with it. I give them props for good DX as well.

Anyway, that’s quickly from me. Would love a DM from you to see what your experiences are. Thank you!

Post #23

kokolegorille

I am using erlport to communicate with python scripts.
I added poolboy to be able to process multiple files concurently, but limited to 10 workers.

Here is an example for the sample file, in french…

iex(9)> WhisperStt.speech_to_text path, "fr"
{:ok,
 %{
   language: "fr",
   segments: [
     %{end: 1.0, start: 0.0, text: "C'est le machine-machin."},
     %{
       end: 3.0,
       start: 1.0,
       text: "La machine-machin est la plus mignonne de l'automobile."
     },
     %{end: 4.0, start: 3.0, text: "Il a des détails très tristiques."},
     %{
       end: 5.0,
       start: 4.0,
       text: "Le tristique, la position, le pain, le poids,"
     },
     %{end: 6.0, start: 5.0, text: "plus un mètre incroyable."},
     %{end: 7.0, start: 6.0, text: "Le mètre machine, le poids, le place,"},
...

I have python installed on the server, and I have a requirements.txt for the plugin.

Post #15