Creating a Digital Human with Membrane, Elixir's video processing toolkit

ElixirConf · June 18, 2025, 1:00pm

Creating a Digital Human with Membrane, Elixir’s video processing toolkit

ElixirConf US 2024 - Łukasz Kita & Nick Dichev

Welcome to ElixirConf
Welcome to ElixirConf

In the landscape of conversational AI, the proliferation of text-based Large Language Model (LLM) chatbots has marked a significant milestone. At Firework, we are pushing the boundaries further by introducing an innovative leap forward: a “digital human.” This AI chatbot transcends traditional interaction paradigms by enabling users to engage through both visual and auditory live streams, offering a more immersive and dynamic user experience.

Central to our breakthrough is a robust backend live streaming server, meticulously engineered using Elixir and leveraging the powerful capabilities of the Membrane framework. This combination not only facilitates seamless real-time interactions but also ensures scalability and efficiency in handling complex audio/video processing tasks.

Our upcoming presentation aims to shed light on the intricate architecture and functionalities of our system, with a special focus on the Membrane framework and its application in Elixir. The session is structured to serve primarily as a comprehensive tutorial on Membrane, guiding attendees through the nuances of audio and video processing pipelines within the context of AI-driven conversational systems.

Key topics include a deep dive into Membrane’s elements—the foundational components of an audio/video processing pipeline. We’ll start with a fundamental overview of how these elements function, followed by a detailed walkthrough of the implementation for a basic yet sophisticated element. Building upon this knowledge, we will explore the advanced elements that constitute the backbone of our “digital human’s” audio/video processing capabilities, offering insights into the practical implementation challenges and solutions.

This presentation is designed to not only showcase our innovative work at Firework but also to empower attendees with the knowledge and tools to explore and innovate within the realm of real-time audio/video processing and conversational AI.

Comments welcome! View the elixirconf tag for more ElixirConf talks!