Cameras (at least the ones I’ve run into) usually have them, but I’m not sure in this particular case. That’s why I suggested @ConnorRigby to print the size of frames. But suppose RTSP source is supplying the desired RTP packets, SPS/PPS will come in single packet which probably translates to a single video frame, is Membrane smart enough to realize they are not picture frames, and wait for the next picture frame to arrive before calling ffmpeg to decode?
xlphs
mat-hek
Membrane Core Team
Membrane just extracts the stream from RTP and passes it to the decoder. That’s a usual situation that PPS and SPS come before picture frames and the decoder seems to handle that well. @ConnorRigby, can you please share some PCAP, so I can have a look?
0
ConnorRigby
Nerves Core Team
From a quick search through RFC 6185, it seems like the sprop-parameter-sets parameter probably has the SPS frames.
Update:
iex(3)> String.split(str, ",")
["Z2QAKaw0yAeAIn5cBbgICAoAAAfSAAHUwdDAAB8/6AAAfP+Fd5caGAAD5/0AAA+f8K7y4b6g",
"aO48MA=="]
iex(4)> String.split(str, ",") |> Enum.map(&Base.decode64!/1)
[
<<103, 100, 0, 41, 172, 52, 200, 7, 128, 34, 126, 92, 5, 184, 8, 8, 10, 0, 0,
7, 210, 0, 1, 212, 193, 208, 192, 0, 31, 63, 232, 0, 0, 124, 255, 133, 119,
151, 26, 24, 0, 3, 231, 253, 0, 0, 15, 159, 240, ...>>,
<<104, 238, 60, 48>>
]
iex(5)> [a, b] = String.split(str, ",") |> Enum.map(&Base.decode64!/1)
[
<<103, 100, 0, 41, 172, 52, 200, 7, 128, 34, 126, 92, 5, 184, 8, 8, 10, 0, 0,
7, 210, 0, 1, 212, 193, 208, 192, 0, 31, 63, 232, 0, 0, 124, 255, 133, 119,
151, 26, 24, 0, 3, 231, 253, 0, 0, 15, 159, 240, ...>>,
<<104, 238, 60, 48>>
]
iex(6)> NAL.Header.parse_unit_header(a)
{:ok,
{%Membrane.Element.RTP.H264.NAL.Header{nal_ref_idc: 3, type: 7},
<<100, 0, 41, 172, 52, 200, 7, 128, 34, 126, 92, 5, 184, 8, 8, 10, 0, 0, 7,
210, 0, 1, 212, 193, 208, 192, 0, 31, 63, 232, 0, 0, 124, 255, 133, 119,
151, 26, 24, 0, 3, 231, 253, 0, 0, 15, ...>>}}
iex(7)> NAL.Header.parse_unit_header(b)
{:ok,
{%Membrane.Element.RTP.H264.NAL.Header{nal_ref_idc: 3, type: 8},
<<238, 60, 48>>}}
iex(8)> {:ok, {header, _}} = NAL.Header.parse_unit_header(a)
{:ok,
{%Membrane.Element.RTP.H264.NAL.Header{nal_ref_idc: 3, type: 7},
<<100, 0, 41, 172, 52, 200, 7, 128, 34, 126, 92, 5, 184, 8, 8, 10, 0, 0, 7,
210, 0, 1, 212, 193, 208, 192, 0, 31, 63, 232, 0, 0, 124, 255, 133, 119,
151, 26, 24, 0, 3, 231, 253, 0, 0, 15, ...>>}}
iex(9)> NAL.Header.decode_type(header)
:single_nalu
iex(10)>
i decoded the data in this parameter and it seems to contain h264 data. I think if this data is sent to ffmpeg before the other data, the stream would work.
@mat-hek is there a way that a source can “skip” stages? This data needs to skip the rtp decoding and jump straight to the h264 depayloader before the stream is started
0
mat-hek
Membrane Core Team
There’s no straightforward way to ‘skip stages’ in membrane. Imho the best option would be to make disector send SPS/PPS via notification to the pipeline, store them in the state and pass to the decoder via options upon spawning. It would require the decoder to support passing SPS/PPS via options, but that should be quite easy to implement.
1
ConnorRigby
Nerves Core Team
Thanks. I’ll try that out. I hacked some things together and i ran into a different error, but let me try building it out the “correct” way first.
0
ConnorRigby
Nerves Core Team
@mat-hek should should pix_fmt be automatically detected somehow? i’ve tried sending the SPS and PPS values to the decoder before the frames come in, but i’m still getting a send_pkt error.
Update:
i’ve managed to get by the send_pkt error, but it looks like my camera’s pixel format is unsupported by membrane_element_ffmpeg currently. the format returned seems to be AV_PIX_FMT_YUVJ420P which is an old format superseded by AV_PIX_FMT_YUV420P which is supported
0
ConnorRigby
Nerves Core Team
another update:
Finally got some video from the camera with some modifications to the ffmpeg element. I’m still getting tons of:
[h264 @ 0x7f2f5c02e080] non-existing PPS 0 referenced
even though i’ve sent (what i believe to be) the pps frame. The video renders and runs smooth, but that message is beyond irritating, so i’d like to fix it if possible. Any ideas?
0
kip
ex_cldr Core Team
Just jumping in to say I’m really enjoying the thread and even more the tenacity by both of your to get this thing going!
6
mat-hek
Membrane Core Team
That error may be printed out by the h264 parser, that is also based on FFmpeg (pure elixir implementation in progress). This can be fixed by passing parameter sets to the parser instead of the decoder.
0