ringods
Sending structured messages containing a binary member: what is copied?
I would like to pass on incoming file upload as separate chunks down a GenStage, but I would like some expert advice/confirmation on my thinking. I also would like to know of anything Elixir might handle differently.
When reading the binary chunks from the Plug connection, I want to create a tuple per chunk. The binary chunk is one of the message tuple fields. When a message is sent from one process to another, the message is copied and then linked into the mailbox of the target process according this and this page.
The documentation on Erlang binaries mention that the real binary data is handled off-heap, with a ProcBin wrapper on-heap.
So, if I create a tuple message containing a binary, am I right that only the tuple with the ProcBin wrapper is copied into the target mailbox and the real binary data is not?
Marked As Solved
rvirding
While that section is of course correct it does manage to hide some of the basics. So small binaries, less than 64 bytes, are stored on the process heap and are copied when sent in messages, while large binaries, greater than 64 bytes, are store off the process heaps in the refc memory and are not copied when sent in messages. This is completely transparent to the user. It also means that Erlang is usable for streaming large amounts of data and has been used for video streaming applications.
There are a few caveats though:
- Seeing binaries in the refc memory can be referenced by many processes it can take a longer time for then to be garbage collected which can cause problems.
- If you make a sub-binary, however small, of a large binary in refc memory then you will keep a reference to the large binary which means that it will not be collected. See binary:copy for a solution.
Everything is a trade-off. 
I prefer to think of refc memory as the large binary space.
Also Liked
michalmuskala
Additionally, starting in OTP 20, “literals” are not copied. Literals are any constant compile-time values - lists, tuples, strings, maps, etc. All components of a tuple/list/map have to be literals as well, so [1, 2] is a literal, while [1, foo] is not.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"