Plugs: Properly calculate Request/Response size in bytes?

deadtrickster · September 12, 2016, 10:25am

Request: headers + body (lazy read?)?
Response: headers + body, but what about files? They are handled by cowboy directly and no after_send callback (well, I added one, but obviously content-length isn’t transferred from cowboy).

OvermindDL1 · September 12, 2016, 2:38pm

Actually sending files is dropped down to a low-level call to the kernel where the linux kernel itself streams the file to the socket after cowboy sends the headers. Cowboy just stats the file for the side. What are you trying to accomplish though?

deadtrickster · September 12, 2016, 3:40pm

Right, cowboy probably uses sendfile, but it must send content-length before. How to intercept it from cowboy?
I’m trying to do exactly what I asked calculate request/response sizes in bytes

OvermindDL1 · September 12, 2016, 4:21pm

Hmm, I’m curious, lets see how it does it.

So the call enters here:

github.com

ninenines/cowboy/blob/5cb2b544b7dcce296bdb956b14521fb036c26ed8/src/cowboy_http.erl#L859


      
          		#stream{version='HTTP/1.1'} ->
          			Size = iolist_size(Data),
          			Transport:send(Socket, [integer_to_binary(Size, 16), <<"\r\n">>, Data, <<"\r\n">>]);
          		#stream{version='HTTP/1.0'} ->
          			Transport:send(Socket, Data)
          	end,
          	maybe_terminate(State, StreamID, Tail, IsFin);
          %% Send a file.
          commands(State=#state{socket=Socket, transport=Transport}, StreamID,
          		[{sendfile, IsFin, Offset, Bytes, Path}|Tail]) ->
          	Transport:sendfile(Socket, Path, Offset, Bytes),
          	maybe_terminate(State, StreamID, Tail, IsFin);
          %% Protocol takeover.
          commands(State0=#state{ref=Ref, parent=Parent, socket=Socket, transport=Transport,
          		opts=Opts, children=Children}, StreamID,
          		[{switch_protocol, Headers, Protocol, InitialState}|_Tail]) ->
          	%% @todo This should be the last stream running otherwise we need to wait before switching.
          	%% @todo If there's streams opened after this one, fail instead of 101.
          	State = cancel_request_timeout(State0),
          	%% @todo When we actually do the upgrade, we only have the one stream left, plus
          	%% possibly some processes terminating. We need a smart strategy for handling the

Which gets passed to the Transport, which for HTTP will be ranch, so that gets passed to here:

github.com

ninenines/ranch/blob/82c1aa77e64c610f6b2ac0666ae5435f5f79fa40/src/ranch_transport.erl#L62-L97


      
          sendfile(Transport, Socket, Filename, Offset, Bytes, Opts)
          		when is_list(Filename) orelse is_atom(Filename)
          		orelse is_binary(Filename) ->
          	ChunkSize = chunk_size(Opts),
          	case file:open(Filename, [read, raw, binary]) of
          		{ok, RawFile} ->
          			_ = case Offset of
          				0 ->
          					ok;
          				_ ->
          					{ok, _} = file:position(RawFile, {bof, Offset})
          			end,
          			try
          				sendfile_loop(Transport, Socket, RawFile, Bytes, 0, ChunkSize)
          			after
          				ok = file:close(RawFile)
          			end;
          		{error, _Reason} = Error ->
          			Error
          	end;

This file has been truncated. show original

The initial call to sendfile in cowboy takes the offset and bytes to send (looks efficient if you want to send only parts of a file, like a resume file header), so lets go ‘up’ the stack:

github.com

elixir-plug/plug/blob/0b387966d2f21cf050ca666f328864b546b4e754/lib/plug/adapters/cowboy/conn.ex#L38-L51


      
          def send_file(req, status, headers, path, offset, length) do
            %File.Stat{type: :regular, size: size} = File.stat!(path)
          
            length =
              cond do
                length == :all -> size
                is_integer(length) -> length
              end
          
            body_fun = fn(socket, transport) -> transport.sendfile(socket, path, offset, length) end
          
            {:ok, req} = Request.reply(status, headers, Request.set_resp_body_fun(length, body_fun, req))
            {:ok, nil, req}
          end

So plug takes the offset and byte size too for cowboy, let’s go higher up to the conn plug itself:

github.com

elixir-plug/plug/blob/v1.2.0/lib/plug/conn.ex#L381


      
          
          It expects a connection that has not been `:sent` yet and sets its
          state to `:sent` afterwards. Otherwise raises `Plug.Conn.AlreadySentError`.
          
          ## Examples
          
              Plug.Conn.send_file(conn, 200, "README.md")
          
          """
          @spec send_file(t, status, filename :: binary, offset ::integer, length :: integer | :all) :: t | no_return
          def send_file(conn, status, file, offset \\ 0, length  \\ :all)
          
          def send_file(%Conn{state: state}, status, _file, _offset, _length)
              when not state in @unsent do
            _ = Plug.Conn.Status.code(status)
            raise AlreadySentError
          end
          
          def send_file(%Conn{adapter: {adapter, payload}, owner: owner} = conn, status, file, offset, length)
              when is_binary(file) do
            conn = run_before_send(%{conn | status: Plug.Conn.Status.code(status), resp_body: nil}, :file)

So plug defaults length to :all, lets follow it back up to the cowboy plug:

github.com

elixir-plug/plug/blob/0b387966d2f21cf050ca666f328864b546b4e754/lib/plug/adapters/cowboy/conn.ex#L43


      
          def send_resp(req, status, headers, body) do
            {:ok, req} = Request.reply(status, headers, body, req)
            {:ok, nil, req}
          end
          
          def send_file(req, status, headers, path, offset, length) do
            %File.Stat{type: :regular, size: size} = File.stat!(path)
          
            length =
              cond do
                length == :all -> size
                is_integer(length) -> length
              end
          
            body_fun = fn(socket, transport) -> transport.sendfile(socket, path, offset, length) end
          
            {:ok, req} = Request.reply(status, headers, Request.set_resp_body_fun(length, body_fun, req))
            {:ok, nil, req}
          end
          
          def send_chunked(req, status, headers) do

And here is where it tests if it is :all and if so it sets the length to the size of the file as from File.stat!.

What are you needing it for? This can actually be an amazingly hard problem because proxies, front-end cachers, etc… can all change the request and headers in a large variety of ways…

deadtrickster · September 13, 2016, 4:18pm

For my Plugs instrumenters: GitHub - deadtrickster/prometheus-plugs: Prometheus.erl Elixir Plugs

Yep, I know some guy actually added these calculations: prometheus-plugs/lib/plug/utils.ex at 478bd29e1c0d5b668d73b596f1378b0ff9e43b1b · zanhsieh/prometheus-plugs · GitHub. Looks like unnecessary to repeat syscall there. But given luck of after_request callback I have no idea what to do.

deadtrickster · October 4, 2016, 8:21am

So looks like I have to revive this thread. I think I can live without file stats but still, how to properly calculate regular request/response sizes (both headers and body)?

OvermindDL1 · October 4, 2016, 2:04pm

I’m still unsure how possible that is. You could calculate the part that Phoenix sends (say putting a plug right after your endpoint maybe?), but even that may not be accurate once it sends out as headers can be added and removed by a lot of things in a lot of ways…

deadtrickster · October 4, 2016, 2:51pm

What I expect(ed): a hook or event to be executed or fired just before the first byte sent. Headers should be already serialized to iolist and all is needed is iolist_size call. This is how I implemented this for Elli: https://github.com/elli-lib/elli/blob/develop/src/elli_http.erl#L595. Possibly it is too low-level for Plugs. And Cowboy is essentially uninstrumentable :(.