marcello
Why is data deep-copied when sending to another process?
Hi everyone,
I’m just learning elixir with the help of “Elixir in Action” by Sasa Juric. In Chapter 5 “Concurrency primitives” there is something I don’t get. Maybe someone can help me?
It’s with regard to deep-copy data when sending data to another process.
“You may wonder about the purpose of shared-nothing concurrency. First, it simplifies the code of each individual process. Because processes don’t share memory, you don’t need complicated synchronization mechanisms such as locks and mutexes. Another benefit is overall stability: one process can’t compromise the memory of another.”
Why should there be a need for sync. mechanisms? Data is immutable, it can’t be mutated by another process. So compromising data shouldn’t be possible.
What am I missing?
Thanks, Marcello
Most Liked
rvirding
I think one thing to realise wrt gc is that if you share data between processes then when you do a gc you have to gc all the processes and the whole heap. This means that you need a real-time collector which complicates things. Running in multiple threads which the BEAM does complicates matters even more as you need to make it thread safe and/or pay a large cost in synchronisation. This is now done for the large binaries which results that sometimes it can take long time to reclaim their memory. Basically every process which has referenced the binary has to do a full gc before the binary can be reclaimed. This can take a comparatively long time and overflowing memory with unreclaimed large binaries is actually a problem.
So while not sharing data and copying data when sending messages is actually quite a good way of doing it even if it sounds wrong. Way back when I did some implementations of erlang with shared memory and real-time collectors and it is not trivial to get it right. Fun though, but not trivial.
In 22 there are some new special memory areas which are shared so some things are good and really fast but you also pay a heavy price. Check out atomics, counters and persistent_terms and you will see what I mean.
rvirding
I think you will end up copying most of the data anyway but it a much less controlled fashion. As someone has pointed out you are viewing the data sent in a message as one big coherent chunk created in the sender and sent to the receiver, but this is generally not the case. Usually the data in a message is put together from smaller bits which come from many different processes. This will mean that either you have a right mess of keeping track of what and from where, or you will be doing a lot of copying between processes which is what you are trying to avoid.
A simple example of what I mean. Assume A sends a chunk of data to B which sends it on to C, so A → B → C. From what I understand of your method is that the data is not copied and A knows it has sent to B and B knows it has sent it to C. What happens if/when B dies? A does not know about C so if A now dies it can’t copy the data to C. Or should B tell A about C? But that means a process has to keep track of where all its data has come from, which we don’t need to do now.
This is exactly the problem you get with atomics, counters and persistent_terms. Atomics and counters solve the problem by only allowing integers while persistent_terms allow complex data structures but require a global gc when you update the data.
hubertlepicki
yes, precisely that. However, what @marcello says is true mostly for the small messages.
If you have a message consisting of binaries larger that 64 bytes (which is, relatively small), these will not be copied and instead will be allocated somewhere in designated space on the BEAM. So, if you are sending large blobs between processes these are not copied over.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security









