How to demux MP4 using membrane framework?

I’m playing around with membrane, trying to stream an MP4(h264/aac) file through HLS to my phone, I think the first thing is to demux the file into video and audio streams. The code look like this

  use Membrane.Pipeline

  def handle_init(path_to_file) do
    children = %{
      source_file: %Membrane.File.Source{location: path_to_file},
      demuxer: Membrane.MPEG.TS.Demuxer,
      video_parser: %Membrane.H264.FFmpeg.Parser{framerate: {24, 1}},
      video_decoder: Membrane.H264.FFmpeg.Decoder,
      audio_decoder: Membrane.AAC.FDK.Decoder,
      audio_converter: %FFmpeg.SWResample.Converter{
        output_caps: %Membrane.Caps.Audio.Raw{channels: 2, format: :s16le, sample_rate: 48_000}
      },
      sink: %Membrane.File.Sink{location: "video_output"},
      portaudio: PortAudio.Sink
    }

    links = [
      link(:source_file) |> to(:demuxer),
      link(:demuxer) |> via_out(Pad.ref(:output, 256)) |> to(:video_parser),
      link(:demuxer) |> via_out(Pad.ref(:output, 257)) |> to(:audio_decoder),
      link(:video_parser) |> to(:video_decoder),
      link(:video_decoder) |> to(:sink),
      link(:audio_decoder) |> to(:audio_converter),
      link(:audio_converter) |> to(:portaudio)
    ]

    spec = %ParentSpec{
      children: children,
      links: links
    }

    {{:ok, spec: spec}, %{}}
  end

However, when I run it, there’s no sound, and also video_output is zero bytes.

iex(3)> Pipeline.play(pid)                              
:ok
iex(4)> 
21:01:41.415 [debug] [pipeline@<0.295.0>] Changing playback state from stopped to prepared
 
21:01:41.441 [debug] [pipeline@<0.295.0>] Playback state changed from stopped to prepared
 
21:01:41.441 [debug] [pipeline@<0.295.0>] Changing playback state from prepared to playing

21:01:41.441 [debug] [:video_decoder] Evaluating playback buffer

21:01:41.441 [debug] [:source_file] Evaluating playback buffer

21:01:41.441 [debug] [:video_parser] Evaluating playback buffer

21:01:41.441 [debug] [:audio_converter] Evaluating playback buffer

21:01:41.441 [debug] [:audio_decoder] Evaluating playback buffer
 
21:01:41.441 [debug] [:sink] Evaluating playback buffer

21:01:41.441 [debug] [:demuxer] Evaluating playback buffer
 
21:01:41.488 [debug] [:portaudio] Evaluating playback buffer
 
21:01:41.488 [debug] [pipeline@<0.295.0>] Playback state changed from prepared to playing
 
21:01:41.499 [debug] [:video_parser] Ignoring event %Membrane.Core.Events.EndOfStream{}
 
21:01:41.499 [debug] [:audio_decoder] Ignoring event %Membrane.Core.Events.EndOfStream{}

I need a bit help here, am I doing this correctly?

1 Like

Hi @hlcfan, you’re right, the first step would be to demux the MP4, but you’re using Membrane.MPEG.TS.Demuxer, that is able to demux MPEG-TS stream, not MP4. We don’t have MP4 demuxer in Membrane yet, though it’s in the roadmap.

3 Likes

Thanks @mat-hek for the quick reply. Looking forward to the MP4 demuxer!

After converting the file to TS format, it works, thanks!

1 Like

@mat-hek Do you have an example of how to stream a TS file through HLS? I can’t seem to find one.

I don’t, but once you have H264 and AAC, you should be able to put them into HLS like in this demo

3 Likes