This is perhaps not what you want to hear, but if the ultimate destination is S3, it’s generally considered a better practice to have the client send the content straight to S3. This is done using signed URLs, which is loosely supported but under-documented in ExAws, removing your server compute from the equation altogether. This approach does require that you’re willing to defer any post-processing that you would ordinarily do in-line during the upload, and to put in the work to make that behavior work asynchronously after the fact.

It’s not universally appropriate for all use cases and if uploads aren’t a core focus of the work in question it may not be worth it, since it can be significantly more engineering time than to just do it naively with the application server, especially if it’s the first time a dev/team is implementing this technique.

I have kind of given up on the stream idea, and instead just a more configurable multipart parser that accepts a module + behavior and some config to allow customizing what happens to the file as the body is being parsed.

I implemented both a Temp and S3 adapter.
Temp is nearly identical to the behavior of the built-in Plug.Upload.
And S3 implements some of the first POC to allow chunk uploading to S3 rather than writing to temp.

I designed it so adapters could implement some custom logic like encryption before writing the file..

It might not be as elegant as using streams would be, but seems to work pretty well!

Yea, I am trying to represent the file parsing as a stream but due to some contention on the conn, or probably that its being accessed in multiple places - this does not seem to work and the stream just hangs. Here is some code:

 Stream.resource(
  fn ->
    IO.inspect("Start Streaming")
    conn
  end,
  fn conn ->
    Plug.Conn.read_part_body(conn, opts)
    |> IO.inspect(label: "read this part!")
    |> case do
      {:ok, tail, conn} -> {tail, conn}
      {:more, tail, conn} -> {tail, conn}
      {:done, conn} -> {:halt, conn}
    end
  end,
  fn _conn ->
    IO.inspect("streaming completed!")
  end
)

The previous attempt going to object storage directly from the parser and blocking the conn worked OK - and behaves more like Plug.Upload, but as you mention N streams to N files inside a Plug.Upload like struct is pretty tricky, and is what I am going for, drawing parallels to nodejs body parsers like busboy

Going lower level may be the only option, but its still just a multipart form we are parsing here, and this would hopefully still work as a simple plug parser…

After looking more at Plug.Conn docs this stands out and is likely the issue i am having:

Because the request body can be of any size, reading the body will only work once, as Plug will not cache the result of these operations. If you need to access the body multiple times, it is your responsibility to store it. Finally keep in mind some plugs like Plug.Parsers may read the body, so the body may be unavailable after being accessed by such plugs.