hubertlepicki
I am interested in using Membrane Framework as a proxy for OpenAI Realtime API and this livebook provided in the examples does almost precisely what I need.
What I am struggling to understand is how this would work in actual production environment.
In particular, I noticed that the pipeline initiated here:
{:ok, _supervisor, pipeline} =
Membrane.Pipeline.start_link(OpenAIPipeline,
openai_ws_opts: openai_ws_opts,
webrtc_source_ws_port: 8829,
webrtc_sink_ws_port: 8831
)
Accepts only one connection, and when this connection drops, it shuts down with:
(ex_webrtc 0.4.1) lib/ex_webrtc/dtls_transport.ex:313: ExWebRTC.DTLSTransport.handle_ice_data/2
(ex_webrtc 0.4.1) lib/ex_webrtc/dtls_transport.ex:287: ExWebRTC.DTLSTransport.handle_info/2
(stdlib 6.0.1) gen_server.erl:2173: :gen_server.try_handle_info/3
(stdlib 6.0.1) gen_server.erl:2261: :gen_server.handle_msg/6
(stdlib 6.0.1) proc_lib.erl:329: :proc_lib.init_p_do_apply/3
Last message: {:ex_ice, #PID<0.992.0>, {:data, <<21, 254, 253, 0, 1, 0, 0, 0, 0, 0, 1, 0, 26, 0, 1, 0, 0, 0, 0, 0, 1, 174, 202, 11, 139, 249, 54, 50, 51, 36, 148, 29, 72, 144, 63, 215, 19, 63, 224>>}}
12:18:34.765 [error] GenServer #PID<0.1051.0> terminating
** (stop) {[], []}
Last message: {:ex_ice, #PID<0.1050.0>, {:data, <<21, 254, 253, 0, 1, 0, 0, 0, 0, 0, 1, 0, 18, 11, 40, 203, 221, 56, 144, 220, 98, 1, 229, 33, 223, 76, 248, 36, 232, 117, 65>>}}
12:18:34.773 [error] <0.325.0>/:webrtc_sink/ Terminating with reason: {:membrane_child_crash, :webrtc, {:timeout_value, [{:gen_server, :loop, 7, [file: ~c"gen_server.erl", line: 2078]}, {ExDTLS, :handle_data, 2, [file: ~c"lib/ex_dtls.ex", line: 168]}, {ExWebRTC.DTLSTransport, :handle_ice_data, 2, [file: ~c"lib/ex_webrtc/dtls_transport.ex", line: 313]}, {ExWebRTC.DTLSTransport, :handle_info, 2, [file: ~c"lib/ex_webrtc/dtls_transport.ex", line: 287]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2173]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2261]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}
12:18:34.773 [error] <0.325.0>/ Terminating with reason: {:membrane_child_crash, :webrtc_sink, {:membrane_child_crash, :webrtc, {:timeout_value, [{:gen_server, :loop, 7, [file: ~c"gen_server.erl", line: 2078]}, {ExDTLS, :handle_data, 2, [file: ~c"lib/ex_dtls.ex", line: 168]}, {ExWebRTC.DTLSTransport, :handle_ice_data, 2, [file: ~c"lib/ex_webrtc/dtls_transport.ex", line: 313]}, {ExWebRTC.DTLSTransport, :handle_info, 2, [file: ~c"lib/ex_webrtc/dtls_transport.ex", line: 287]}, {:gen_server, :try_handle_info, 3, [file: ~c"gen_server.erl", line: 2173]}, {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 2261]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 329]}]}}}
Is this the expected behavior?
If yes, then how would I use that in production? Should I reserve a bunch of ports, and start a pool of pipelines and assign my clients to them?
What about authentication of users, I guess I would need to generate the token that they would pass in the websocket URL but how do I hook up the code server-side to validate it?
Any hints much appreciated <3
Where Next?
Trending in Discussions
Other Trending Topics
Latest Membrane Threads
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixirconf-us
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
Feliks
Hi Hubert!
You are right, one
MembranePipeline is able to handle only one client at the time, so you would have to reserve a bunch of ports, to be able to handle multiple clients at the time.But, I am currently working on
LiveViewcomponents that will be able to exchange WebRTC signaling messages via LiveView WebSocket, so using them should remove the need of reserving many ports. I can let you know, when they will be ready.BTW, we also developed similar demo that uses Boombox
This demo works even better, because it allows you to interrupt the chat, while it speaks.
Boomboxspawns Membrane Pipeline under the hood, so it if Membrane Pipeline works well in your case, Boombox should too, and it has simpler API.I think that using LiveView components that I have mentioned would also solve the problem with authentication, since the authentication provided by Phoenix application makes the websocket used by LiveView secure
hubertlepicki
Hmm I got some error when I tried to run the Boombox example. I’ll report the details when I’m again in front of that machine that I tried it on. But thanks!
nefty
I am also making a similar project and had the same question as you. I managed to get it working with Membrane WebRTC plugin and Membrane.WebRTC.SignalingChannel. I use the SignalingChannel to pass the messages to the LiveView, which then sends the messages to the JS Hook as events. Here is a simple repo showing a LiveView version of Echo, which just receives the audio and video from the client and sends it back unchanged, using a LiveView and Membrane WebRTC. You can add any Membrane elements in the middle of the pipeline to communicate with OpenAI, etc.
The repo is messy still but it should give you an idea. This can obviously be extended to mulitple listeners with PubSub, like in the LiveBroadcaster example, but I haven’t done that yet. It would just be a matter of publishing the SignalingChannel messages to the right PubSub channel, and your LiveViews listening on that channel.
Here is the repo:
https://github.com/nefty/live_echo
If you have any questions, let me know and I’ll try to explain, but I am still learning Membrane and ExWebRTC so I will do my best. Here is a simple diagram of the architecture as wellL
druyang
I don’t mean to pester, but this seems like a really useful project to enable membrane for liveview. Are you still working on it?
druyang
Still learning RTC and membrane, so forgive my ignorance. Would the liveview app be responsible for “routing” the ExWebRTC messages to the “correct” membrane pipeline?
For example, if each group of users has an exclusive openAI realtime connection, the liveview server would be responsible for creating and publishing to the respective membrane pipeline instance for each group?
Feliks
Yes, it would be liveview’s responsibility
Feliks
It is available now there: GitHub - membraneframework-labs/membrane_webrtc_live at implement-webrtc-components · GitHub
Feel free to use this branch as a dependency in your project, the code there should already work fine. Currently I am waiting on the review from a person that is now on the holidays, so IDK when I will merge it.
Probably in the long run these liveview components will be moved to
:membrane_webrtc_pluginpackage, so if you discover one day that this repository doesn’t exist anymore, look for new version ofmembrane_webrtc_pluginor at least its repository inmembraneframeworkorganisation on GitHub.In case of questions feel free to ask
BTW LiveViews from the linked repo solve the problem of which WebRTC signaling messages should be sent to which Membrane Elements
Feliks
@druyang I have merged the PR so the linked branch is deleted now
Feliks
@druyang @hubertlepicki LiveViews for Membrane WebRTC have been released in new version of
membrane_webrtc_pluginThese modules depend on
phoenixandphoenx_live_viewwhich are optional dependencies inmembrane_webrtc_plugin, so you have to add these two deps to your project.In case of any issues, please let me know.
hubertlepicki
I will test these but probably only next week. Been snowed under work this week, sorry guys