As a filter, I think. So that you can connect a TCP source to the input and RTP bin to the output.
mat-hek
Membrane Core Team
entone
ConnorRigby
Nerves Core Team
I tried adding a single filter between the TCP socket and the RTP bin:
defmodule Disect do
use Membrane.Filter
alias Membrane.Buffer
def_input_pad :input, caps: :any, demand_unit: :buffers
def_output_pad :output, caps: :any
@impl true
def handle_init(state) do
state = Map.merge(state || %{}, %{
buffer: <<>>
})
{:ok, state}
end
@impl true
def handle_demand(:output, size, :buffers, _ctx, state) do
{{:ok, demand: {:input, size}}, state}
end
@impl true
def handle_process(:input, %{payload: payload}, _ctx, state) do
disect(state.buffer <> payload, [], state)
end
def disect(<<36, 0::integer-8, length::integer-16, chunk::binary-size(length), rest::binary>>, acc, state) do
# chunk is an RTP packet, forward to RTP parser
disect(rest, acc ++ [%Buffer{payload: chunk}], state)
end
def disect(<<36, channel::integer-8, length::integer-16, _chunk::binary-size(length), rest::binary>>, acc, state) do
IO.puts "ignoring channel: #{channel}"
disect(rest, acc, state)
end
# ignoring non video packets for simplicity
def disect(<<36, _::binary>> = incomplete, acc, state) do
# need to request more data
{{:ok, [buffer: {:output, acc}, redemand: :output]}, %{state | buffer: incomplete}}
end
def disect(<<>>, acc, state) do
{{:ok, [buffer: {:output, acc}, redemand: :output]}, %{state | buffer: <<>>}}
end
end
but i’m still getting a similar error to what i received before:
09:00:54.691 [error] GenServer #PID<0.403.0> terminating
** (Membrane.ActionError) Error while handling :split action:
Unknown error: :send_pkt
Callback: Membrane.Element.FFmpeg.H264.Decoder.handle_process_list
Action args: {:handle_process,
[
[
:input,
%Membrane.Buffer{
metadata: %{},
payload: <<0, 0, 0, 1, 101, 136, 128, 16, 0, 12, 255, 245, 154, 34, 103,
162, 245, 12, 56, 225, 60, 222, 189, 150, 153, 78, 16, 77, 254, 201,
165, 53, 240, 253, 3, 133, 170, 112, 0, 254, 178, 211, 19, ...>>
}
]
]}
The issue i have here is that in my wireshark dump, those bytes don’t exist in any of the packets. I have wireshark pcap if you are interested. Any ideas?
UPDATE:
i found this line that adds the first 4 bytes onto the head of that paylod, but i still can’t find even 101, 136, 128, 16 or any other portion of this paylod in the pcap.
0
xlphs
The payload <<0, 0, 0, 1, 101, 136, ... is H264 frame that has been pieced together from RTP packets, the RFC for H264 over RTP explains the process, basically you have to take apart the first and second byte to get the fragment type and NAL type then put them together, so you will not find the first couple bytes from RTSP dump.
It’s been some time since I wrote RTSP client, but I think the disect method needs to double check it has actually got length number of bytes. And since you are dealing with interleaved stream, you should not ignore non video packets like that, I think what happens is erlang tcp buffer size is usually bigger than 1 RTSP packet, so you will get multiple RTSP packets every time, which means you need to code the logic to split up RTSP packets.
0
ConnorRigby
Nerves Core Team
Thanks for the info. My dissect function is ensuring that the packets lengths are correct as far as i can tell. Everything from {:tcp, data} is buffered and is able to be parsed successfully as RTP packets i think. The recursive function takes care of this: (i’ve added more comments)
# complete packet verified length
def disect(<<36, 0::integer-8, length::integer-16, chunk::binary-size(length), rest::binary>>, acc, state) do
# chunk is an RTP packet, forward to RTP parser
# rest should be the beginning of the next packet, it may not be complete
disect(rest, acc ++ [%Buffer{payload: chunk}], state)
end
# this has never actually happened in my tests because my camera is currently configured to only output
# one channel currently.
def disect(<<36, channel::integer-8, length::integer-16, _chunk::binary-size(length), rest::binary>>, acc, state) do
# _chunk is a RTP packet, but we don't need it
IO.puts "ignoring channel: #{channel}"
disect(rest, acc, state)
end
# this packet is not complete. Stop recursion, dispatch complete packets, buffer the incomplete data
def disect(<<36, _::binary>> = incomplete, acc, state) do
# need to request more data
{{:ok, [buffer: {:output, acc}, redemand: :output]}, %{state | buffer: incomplete}}
end
0
xlphs
If you buffer the incomplete packet… the next chunk of tcp bytes is not going to start with the RTSP magic code 36, and how would you know how many bytes left to read for this incomplete packet?
Edit: thanks for the added comment I see now your code should work.
It seems now the problem is decoding h264? I suggest you print the size of your frames, H264 should have a few small frames (SPS, PPS, etc) at the beginning before the IDR frame, followed by many smaller P frames. If you don’t have the SPS/PPS frames, that’s what the “non-existing PPS 0 referenced” error is about.
0
ConnorRigby
Nerves Core Team
I think i must be missing something, so i made a really dumbed down receiver for RTP data. I tested this out just now with wireshark and every RTP frame that wireshark sees, this code sees. I compared the RTP packet sequence_number and they are in order and complete.
# start here
def run() do
# imagine rtsp SETUP and PLAY request here
socket = do_rtsp_setup_stuff()
# enter the buffering loop
loop(socket, <<>>)
end
def loop(socket, old_buffer_from_last_disect) do
# get new unparsed data from the socket
data = recv(socket)
# disect the data starting with the unparsed data from last loop
{complete_packets, incomplete_packets} = disect(old_buffer_from_last_disect <> data, [])
loop(socket, unparsed_data)
end
# complete packet verified length
def disect(<<36, 0::integer-8, length::integer-16, chunk::binary-size(length), rest::binary>>, complete_packets) do
# chunk is an RTP packet, forward to RTP parser
# rest should be the beginning of the next packet, it may not be complete
packet = RTP.decode!(chunk)
disect(rest, complete_packets ++ [packet])
end
# this has never actually happened in my tests because my camera is currently configured to only output
# one channel currently.
def disect(<<36, channel::integer-8, length::integer-16, _chunk::binary-size(length), rest::binary>>, complete_packets) do
# _chunk is a RTP packet, but we don't need it
IO.puts "ignoring channel: #{channel}"
disect(rest, complete_packets)
end
# this packet is not complete. Stop recursion, dispatch complete packets, buffer the incomplete data
def disect(<<36, _::binary>> = incomplete_packets, complete_packets) do
# need to request more data
{complete_packets, incomplete_packets}
end
0
xlphs
I have a vague feeling your issue is Membrane is not prepending SPS and PPS in front of IDR frame. It is a weird thing with ffmpeg, I have had many headaches with this before. Just a thought, I’m probably wrong as I have basically no working knowledge of Membrane.
1
ConnorRigby
Nerves Core Team
I don’t know enough about ffmpeg or even h264 to have a lot of input on this, but while watching what VLC does with wireshark, it seems to open the stream stream (DESCRIBE, SETUP, PLAY, etc), and data starts flowing in, but VLC doesn’t display anything for quite a while. I assume this is waiting for keyframes(?).
UPDATE: i just googled what SPS and PPS was and found some more information: on stack overflow
0