I don’t know enough about ffmpeg or even h264 to have a lot of input on this, but while watching what VLC does with wireshark, it seems to open the stream stream (DESCRIBE, SETUP, PLAY, etc), and data starts flowing in, but VLC doesn’t display anything for quite a while. I assume this is waiting for keyframes(?).
UPDATE: i just googled what SPS and PPS was and found some more information: on stack overflow
Cameras (at least the ones I’ve run into) usually have them, but I’m not sure in this particular case. That’s why I suggested @ConnorRigby to print the size of frames. But suppose RTSP source is supplying the desired RTP packets, SPS/PPS will come in single packet which probably translates to a single video frame, is Membrane smart enough to realize they are not picture frames, and wait for the next picture frame to arrive before calling ffmpeg to decode?
Membrane just extracts the stream from RTP and passes it to the decoder. That’s a usual situation that PPS and SPS come before picture frames and the decoder seems to handle that well. @ConnorRigby, can you please share some PCAP, so I can have a look?
@mat-hek here is a couple of pcaps. The file vlc is a successful stream, membrane is the membrane stream.
From a quick search through RFC 6185, it seems like the sprop-parameter-sets parameter probably has the SPS frames.
Update:
iex(3)> String.split(str, ",")
["Z2QAKaw0yAeAIn5cBbgICAoAAAfSAAHUwdDAAB8/6AAAfP+Fd5caGAAD5/0AAA+f8K7y4b6g",
"aO48MA=="]
iex(4)> String.split(str, ",") |> Enum.map(&Base.decode64!/1)
[
<<103, 100, 0, 41, 172, 52, 200, 7, 128, 34, 126, 92, 5, 184, 8, 8, 10, 0, 0,
7, 210, 0, 1, 212, 193, 208, 192, 0, 31, 63, 232, 0, 0, 124, 255, 133, 119,
151, 26, 24, 0, 3, 231, 253, 0, 0, 15, 159, 240, ...>>,
<<104, 238, 60, 48>>
]
iex(5)> [a, b] = String.split(str, ",") |> Enum.map(&Base.decode64!/1)
[
<<103, 100, 0, 41, 172, 52, 200, 7, 128, 34, 126, 92, 5, 184, 8, 8, 10, 0, 0,
7, 210, 0, 1, 212, 193, 208, 192, 0, 31, 63, 232, 0, 0, 124, 255, 133, 119,
151, 26, 24, 0, 3, 231, 253, 0, 0, 15, 159, 240, ...>>,
<<104, 238, 60, 48>>
]
iex(6)> NAL.Header.parse_unit_header(a)
{:ok,
{%Membrane.Element.RTP.H264.NAL.Header{nal_ref_idc: 3, type: 7},
<<100, 0, 41, 172, 52, 200, 7, 128, 34, 126, 92, 5, 184, 8, 8, 10, 0, 0, 7,
210, 0, 1, 212, 193, 208, 192, 0, 31, 63, 232, 0, 0, 124, 255, 133, 119,
151, 26, 24, 0, 3, 231, 253, 0, 0, 15, ...>>}}
iex(7)> NAL.Header.parse_unit_header(b)
{:ok,
{%Membrane.Element.RTP.H264.NAL.Header{nal_ref_idc: 3, type: 8},
<<238, 60, 48>>}}
iex(8)> {:ok, {header, _}} = NAL.Header.parse_unit_header(a)
{:ok,
{%Membrane.Element.RTP.H264.NAL.Header{nal_ref_idc: 3, type: 7},
<<100, 0, 41, 172, 52, 200, 7, 128, 34, 126, 92, 5, 184, 8, 8, 10, 0, 0, 7,
210, 0, 1, 212, 193, 208, 192, 0, 31, 63, 232, 0, 0, 124, 255, 133, 119,
151, 26, 24, 0, 3, 231, 253, 0, 0, 15, ...>>}}
iex(9)> NAL.Header.decode_type(header)
:single_nalu
iex(10)>
i decoded the data in this parameter and it seems to contain h264 data. I think if this data is sent to ffmpeg before the other data, the stream would work.
@mat-hek is there a way that a source can “skip” stages? This data needs to skip the rtp decoding and jump straight to the h264 depayloader before the stream is started
There’s no straightforward way to ‘skip stages’ in membrane. Imho the best option would be to make disector send SPS/PPS via notification to the pipeline, store them in the state and pass to the decoder via options upon spawning. It would require the decoder to support passing SPS/PPS via options, but that should be quite easy to implement.
Thanks. I’ll try that out. I hacked some things together and i ran into a different error, but let me try building it out the “correct” way first.
@mat-hek should should pix_fmt be automatically detected somehow? i’ve tried sending the SPS and PPS values to the decoder before the frames come in, but i’m still getting a send_pkt error.
Update:
i’ve managed to get by the send_pkt error, but it looks like my camera’s pixel format is unsupported by membrane_element_ffmpeg currently. the format returned seems to be AV_PIX_FMT_YUVJ420P which is an old format superseded by AV_PIX_FMT_YUV420P which is supported
another update:
Finally got some video from the camera with some modifications to the ffmpeg element. I’m still getting tons of:
[h264 @ 0x7f2f5c02e080] non-existing PPS 0 referenced
even though i’ve sent (what i believe to be) the pps frame. The video renders and runs smooth, but that message is beyond irritating, so i’d like to fix it if possible. Any ideas?
Just jumping in to say I’m really enjoying the thread and even more the tenacity by both of your to get this thing going!
That error may be printed out by the h264 parser, that is also based on FFmpeg (pure elixir implementation in progress). This can be fixed by passing parameter sets to the parser instead of the decoder.
ah i see. I’ll have to add that to another PR
Tried that and it at least changed the messages.
[h264 @ 0x7f78d002a7c0] Frame num change from 0 to 1
[h264 @ 0x7f78d002a7c0] decode_slice_header error
[h264 @ 0x7f78d002a7c0] error while decoding MB 15 1, bytestream -20
[h264 @ 0x7f78d002a7c0] concealing 8074 DC, 8074 AC, 8074 MV errors in P frame
[h264 @ 0x7f78d002a7c0] error while decoding MB 39 1, bytestream -6
[h264 @ 0x7f78d002a7c0] concealing 8050 DC, 8050 AC, 8050 MV errors in P frame
[h264 @ 0x7f78d002a7c0] error while decoding MB 19 1, bytestream -12
[h264 @ 0x7f78d002a7c0] concealing 8070 DC, 8070 AC, 8070 MV errors in P frame [h264 @ 0x7f78d002a7c0] error while decoding MB 21 1, bytestream -8
Are you sure that parameter sets are still passed to the decoder also?
yes they are passed to both in the same order, sps first, pps second. The decoder will error and fail completely if i don’t send both sps and pps. The parser seems to be okay without either, but sends that non-existing PPS 0 referenced error on every frame. Adding the sps and pps to the parser resulted in those errors from my last message. In either case, the video stream still works correctly
You are getting video as expected but the PPS error comes with every frame? It sounds like the decoder is now overzealous in making sure sps/pps are present 
yes i’m getting valid video, but every RTP frame that is parsed/decoded logs that non-existing PPS 0 referenced error. Like nothing bad seems to be happeneing, just an annoying log.
FFmpeg has an option to silence logs by changing log level, maybe try that.
These logs mean that the decoder is able to parse all the headers and metadata, but some parts of the stream content (actual video) are malformed/incompatible. How often is that printed? I suppose it might have occurred before, but you didn’t see it because of the amount of non-existing PPS 0 referenced logs. Also, make sure that parameter sets are not duplicated - both provided to the decoder via options and forwarded there by the parser (the second way is the best solution imho).
Silencing logs may not be the best way to go, as they can possibly carry some useful information in case of failure.
It prints it pretty constantly. I haven’t checked exactly but it looks like it happens on every frame. These logs weren’t here before i sent the sps and pps values to the parser. I’ll double check that it’s being sent/forwarded correctly.
© Copyright Elixir Forum | Terms | Privacy & Cookies
