Plugs: Properly calculate Request/Response size in bytes?

Request: headers + body (lazy read?)?
Response: headers + body, but what about files? They are handled by cowboy directly and no after_send callback (well, I added one, but obviously content-length isn’t transferred from cowboy).

Actually sending files is dropped down to a low-level call to the kernel where the linux kernel itself streams the file to the socket after cowboy sends the headers. Cowboy just stats the file for the side. What are you trying to accomplish though?

Right, cowboy probably uses sendfile, but it must send content-length before. How to intercept it from cowboy?
I’m trying to do exactly what I asked :slight_smile: calculate request/response sizes in bytes :slight_smile:

Hmm, I’m curious, lets see how it does it.

So the call enters here:
https://github.com/ninenines/cowboy/blob/5cb2b544b7dcce296bdb956b14521fb036c26ed8/src/cowboy_http.erl#L859

Which gets passed to the Transport, which for HTTP will be ranch, so that gets passed to here:
https://github.com/ninenines/ranch/blob/82c1aa77e64c610f6b2ac0666ae5435f5f79fa40/src/ranch_transport.erl#L62-L97

The initial call to sendfile in cowboy takes the offset and bytes to send (looks efficient if you want to send only parts of a file, like a resume file header), so lets go ‘up’ the stack:
https://github.com/elixir-lang/plug/blob/0b387966d2f21cf050ca666f328864b546b4e754/lib/plug/adapters/cowboy/conn.ex#L38-L51

So plug takes the offset and byte size too for cowboy, let’s go higher up to the conn plug itself:
https://github.com/elixir-lang/plug/blob/v1.2.0/lib/plug/conn.ex#L381

So plug defaults length to :all, lets follow it back up to the cowboy plug:
https://github.com/elixir-lang/plug/blob/0b387966d2f21cf050ca666f328864b546b4e754/lib/plug/adapters/cowboy/conn.ex#L43

And here is where it tests if it is :all and if so it sets the length to the size of the file as from File.stat!. :slight_smile:

What are you needing it for? This can actually be an amazingly hard problem because proxies, front-end cachers, etc… can all change the request and headers in a large variety of ways…

For my Plugs instrumenters: https://github.com/deadtrickster/prometheus-plugs

Yep, I know some guy actually added these calculations: https://github.com/zanhsieh/prometheus-plugs/blob/478bd29e1c0d5b668d73b596f1378b0ff9e43b1b/lib/plug/utils.ex. Looks like unnecessary to repeat syscall there. But given luck of after_request callback I have no idea what to do.

So looks like I have to revive this thread. I think I can live without file stats but still, how to properly calculate regular request/response sizes (both headers and body)?

I’m still unsure how possible that is. You could calculate the part that Phoenix sends (say putting a plug right after your endpoint maybe?), but even that may not be accurate once it sends out as headers can be added and removed by a lot of things in a lot of ways…

What I expect(ed): a hook or event to be executed or fired just before the first byte sent. Headers should be already serialized to iolist and all is needed is iolist_size call. This is how I implemented this for Elli: https://github.com/elli-lib/elli/blob/develop/src/elli_http.erl#L595. Possibly it is too low-level for Plugs. And Cowboy is essentially uninstrumentable :(.