Erlang 21.2 erts-10.2 released

OvermindDL1

A few good fixes and enhancements, SSL added more methods and is faster, socket polling is faster, however 2 big things that I like!

New counters and atomics modules supplies access to highly efficient operations on mutable fixed word sized variables.

The new :atomics module wraps the low level hardware atomic instructions without any locking, it’s use-cases are limited but EXTREMELY useful for where they are useful. I find it odd how the array part is 1-indexed though, erlang has a lot of mixed 0-index and 1-index code, and 1-indexed means the math is harder… blah…

The new :counters module uses the new :atomics module to do ETS-style counter handling but using hardware atomic instructions so there is no locking or cost. Same thing with the really weird non-mathematical 1-indexed arrays though.

New module persistent_term!. Lookups are in constant time! No copying the terms!

You know the global module that discord put out that recompiles a module to bake terms into it for super-fast referencing and no-copies and so forth? That’s basically the new erlang persistent_term library! Here’s a use session in Elixir!

iex(1)> :persistent_term.info()                        
%{count: 0, memory: 152}
iex(2)> :persistent_term.put(:blah, 42)
:ok
iex(3)> :persistent_term.info()        
%{count: 1, memory: 192}
iex(4)> :persistent_term.get(:blah)
42
iex(5)> :persistent_term.get()     
[blah: 42]
iex(6)> :persistent_term.put(:blorp, make_ref())
:ok
iex(7)> :persistent_term.get()                  
[blah: 42, blorp: #Reference<0.3525248772.1957167107.195526>]
iex(8)> :persistent_term.erase(:blah)
true
iex(9)> :persistent_term.get()                  
[blorp: #Reference<0.3525248772.1957167107.195526>]
iex(10)> :persistent_term.info()
%{count: 1, memory: 216}

So it’s slow to put but crazy-fast to get, reference, and pass around the values stored within! In general you shouldn’t use this unless you know what you are doing as there is no garbage collection, should generally be used for global settings and data, etc…

14 comments

#erlang #erlang-news

24 4131 14

2019-09-18 22:12:34 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

Nicd

These are cool, thanks for the examples! I use ETS to store my blog’s data in memory, think I could use that storage instead? Does it have atomic updates of terms? Then I would be really interested.

Post #1

OvermindDL1

Think of it this way: If it’s something you would normally compile ‘into’ a models code for efficiency then use it, else you probably want ETS (or mnesia to keep it serialized to disk and hold it in memory on load).

So for a blog I’d personally use either flat-files on disk or store it in mnesia with disk and memory copies, I’m not sure I’d use this for that, though your data is static enough that it ‘should’ be fine.

Just remember, :persistent_term has a max storage size of 1 gig unless you set the _MIscs 2048 or so option (that one would raise it to 2 gigs, the value is in megabytes).

When a new value is put then it is visible everywhere immediately on the node because when an old value is replaced or erased then a global garbage collection is run, which can pause things, that collection will replace any links to the global data in all processes to a local copy of the data instead.

Just remember that accessing data in :persistant_term is way fast, but changing data in any way is very slow, and in some cases it can be really really slow. But it’s still faster than recompiling a module to ‘intern’ the terms (which would destroy linked processes if it got updated, so :persistent_term is definitely better there).

Post #2

Nicd

How slow is slow in this case? Any rough estimates?

In my use case, all the blog posts are stored in the filesystem so persistence is not an issue. ETS works fine for serving them from memory, but I’m enticed by this API that is much simpler (ETS is just awful in that regard).

BTW is Application env similar? It has put_env that is global, how is that implemented? Don’t worry, I’m not planning on abusing it for this.

Post #3

OvermindDL1

Depends on many factors, like how many terms are in the persistent cache, how many processes are running in the system, how many modules are loaded, etc… etc… On average except up to a second, some cases could potentially be a few seconds, on an empty system with about nothing running in the BEAM it will be milliseconds I’d think.

Application env’s are in ETS (I think?).

Post #4

michalmuskala

Changing anything in persistent_term means running a GC over all the processes in the system and copying all of the data in persistent_term. It can get very expensive on bigger systems.

Post #5

Nicd

This would totally work for my blog then. I update the data maybe once a month at most (when I write a post) and the only thing running are the Raxx request handlers.

Post #7

michalmuskala

One thing to ask is, if it would be actually beneficial. Binaries (and I assume your entries are just plain binaries) are stored off-heap, so they are not copied when retrieved from ETS anyway. This removes the biggest advantage of persistent_term over plain old ETS.

Post #8

Nicd

Ah very good point. Maybe I’ll just forget about it for this use case. Thanks for all the responses.

Post #9

kip

ex_cldr Core Team

Are their any anecdotes yet reflecting relative performance of :persistent_term versus functions that bake in data?

:persistent_term looks like a possible good approach for storing locale data in my ex_cldr package since its large and static. However it is a fairly complex Map.t() so their would be some penalty to pay in keeping the data as one large map as apposed to the current approach which decomposes the map into different functions.

Post #10