jschaeff

jschaeff

Passing binary data through functions

Hello,

TL;DR: is it OK in elixir to pass large (MBytes) binary strings through function parameters ? If not, what would be a clean way to transfer this binary to the end user in a Plug HTTP response ?

Context: My program reads binary data from storage using a NIF, in a format that Elixir does not know about. Basically, this binary string has to be delivered to the end user through HTTP request. It can get quite big (10s of MBytes) and it works in my demonstrator by passing the binary string through the return of the NIF function and then passed along to the caller function which would then send it to the client through Plug.Connection.send/2

case MSeed.NIF.stream_data(conn.assigns.datafiles) do
  {:ok, msrecords} -> send_data(conn, msrecords)
end

And the send_data/2 function prepares the HTTP result (resp headers) and sends the content of msrecords, like:

Plug.Conn.send_resp(conn, 200, msrecords)

What are the impacts of passing large data through function parameters ? Are there other better ways to do this ?

Marked As Solved

Asd

Asd

It is

None

Also Liked

christhekeele

christhekeele

Additionally, these shared references are refcounted by processes that have ever accessed them.

So it is safe and cheap to pass them through function parameters, and safe and cheap to send them between processes, but dangerous to handle them in long-running processes as the long-lived process will cause the reference to be held, filling up RAM with your large binary, which cannot be reclaimed without manual intervention. At least not until every process that has ever handled them has died. For a recent thread on this phenomenon with detection, remediation, and other tips check out Agent keeps large binary memory despite no state .

If the process reading from your NIF is short-lived, you will be fine; if it’s a GenServer you may want to spin up a worker process on demand to do the actual handling to prevent memory bloat. Phoenix uses short-lived processes per-request so that process boundary should be fine to handle large binaries.

jschaeff

jschaeff

Thanks for all the answers, short and detailed ones.

Indeed, my NIF function is a short-living one, I think I’m safe here.

Again, this forum is great thanks to you all nice folks. Have a nice day !

NobbZ

NobbZ

Large binaries (> 64 byte or so) are passed by reference. Small binaries by value.

Only exception, the small binarie is a substring of a large binarie, in that case a fat pointer is passed, hindering the original binary from garbage collection as long as the substring is “alive”, thats why many slicing operations have a :copy option.

The only “expensive” action is to send them over a network.

If we are talking about large nested data structures (maps + lists) then sending them across processes might already be expensive, as this involves a deep copy!

Binaries are the only datatype that exists on a “shared” heap.

Where Next?

Popular in Questions Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
earth10
Hi, I’m just starting to build a side-project with Elixir and Phoenix and doing some basic test with Elixir alone. What strikes me is th...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
jaysoifer
Is there a way to rollback a specific migration and only that one (“skipping” all the other ones)? Would mix ecto.rollback -v 200809061...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lists...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New

Other popular topics Top

skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31142 143
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
marick
I had some trouble figuring out how to make many-to-many associations work. Once I got it working, I wrote a blog post. Because I’m a nov...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement