Does Elixir perform well for streaming S3 file content to clients?

Right now we’re exploring setting up a API endpoint to stream S3 file content to users. Is Elixir a good fit for this feature, will it perform well or are there better alternatives?

If it’s good enough, I’d like to stay within Elixir land instead of reaching for a different language.

Elixir can do very well at this, although I would also consider simply using presigned S3 URLs so that you can just offload the bandwidth onto S3 directly.

We’re using that already right now on prod. Works well but we’re exploring a different approach.

Do you have any recommended reading for this @benwilson512?

What is the performance story for https://hexdocs.pm/plug/Plug.Conn.html#send_file/3

Any special considerations you have needed to take to scale this?

That function generally expects a file on disk. You’d need to download the whole file from S3, write it to disk, then do send_file.

Can you talk about what you’re trying to achieve by proxying S3? It’s hard to suggest a design without knowing the design goals.

You’d need to download the whole file from S3, write it to disk, then do send_file.

It also should be possible to accept chunks of data in cowboy and send it (also in chunks, just rewriting the headers) to aws without saving anything to disc. Not sure if it is possible with plug.

This way the overhead would be minimized, probably.

1 Like