wmnnd
High memory consumption when uploading files with Phoenix
Hi there,
a little while ago, @josevalim pointed out in this thread that Phoenix is well-suited for handling large uploads since it doesn’t load the whole file into memory but instead writes it to disk instead.
However, I have noticed (using Elixir 1.4.5, Erlang/OTP 20 and Phoenix 1.3.0-rc.2) that uploads of large files (as multipart/form-data, i. e. using Plug.Parsers.MULTIPART) can cause a huge increase in memory consumption:
A minimal Phoenix application reports 35 MB memory usage in Erlang Observer. When uploading a file, this increases to over 100 MB per upload. This is not affected by upload speed and happens both with a throttled 1 MB/s connection and unthrottled uploads to localhost.
Once the connection has been terminated, the memory consumption goes down again to its original value.
Here is the Observer load chart for three subsequent 1 GB uploads to localhost:
So while there is no long-term memory leaking here, could it be that there is a temporary memory leak in Plug.Parsers that gets cleaned up when the connection is terminated?
If you want to quickly try this out, I have created a minimal Phoenix app with Observer and an upload form here:
https://github.com/wmnnd/phoenix-upload-test-case
Most Liked
josevalim
wmnnd
TLDR: No matter which arguments you pass to Plug.Parsers, Plug.Conn will always read and write to disk chunks of 8 MB.
I’ve spent all afternoon trying different things but I think I have found one main cause of the problem: Ultimately, the :read_length setting has no effect because the naming of the opts getting passed around are getting confusing.
Plug.Conn.read_part_body/8 has a guard clause to check whether the chunk of data that has been read is larger than the length parameter. This length parameter gets extracted from the opts Keyword list passed to Plug.Conn.read_part_body/3. However, if Plug.Conn.read_part_body/3 gets called from Plug.Parsers.MULTIPART, :length parameter will always be unset because Plug.Parsers.MULTIPART removes it in order to use it for evaluating the total file size limit.
Here is where it gets even more fun: It doesn’t even matter that length is not passed on because when Plug.Conn.read_part_body/8 calls next/multipart/3, it doesn’t pass this length. Instead, it only passes on the opts Keyword list (without :length). It seems that adapter.read_req_body/2 doesn’t know the :read_length option and then defaults to reading 8_000_000 bytes anyways - which renders the guard check for the chunk size in Plug.Conn.read_part_body/8 useless.
wmnnd
Popular in Questions
Other popular topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"