Best way to deal with immutability and to keep in memory a large list of structs where just 2 fields are frequently changed

Questions / Help
1

I have to keep a large list of points in memory for real time access, after fetching the table from a DB.

Each point is a struct that has many fields, but just two fields will be changing: value and quality.

This list can grow over 100 thousand points.

The changes will occur every 1 second. And can affect many points, hundreds or even thorusands.

Since list is immutable, I can´t change an item in the list.

To avoid rebinding the whole list every second, I need to find a way to be able to deal with immutability.

I thought about keeping 3 parallel lists: 1 containg the fixed fields, 1 containing just the value field, 1 for the quality.

But that will let the design somehow awkward, cause I will have to do first a lookup in the fixed list to find the right point based on a address, and use that index position to change either the corresponding value and/or qualitty.

There must be a clever or simpler way to design it.

Hope someone has deal with such a problem.

Thanks for the attention.

1 Like
2

Well, this is almost exactly what ETS is designed to solve? You can also use Elixir wrappers like Cachex and Ane.

3 Likes
3

It sounds like what you want is a map, not a list. If you are doing index based updates you can do a map where the index is the key, and the value is the value. Updates to maps are memory efficient and do not require rebuilding the entire map.

:ets is another good choice as mentioned by @dimitarvp

1 Like
4

Thanks dimitarvp and benwilson512 for the quick reply.

Even thought I found good resources regarding using ETS:
[Erlang Term Storage (ETS) · Elixir School]

It was straight forward using Cachex.

Ane uses one-based index, which for me does not work since I need a key made up of 2 fields.

Cachex gives the flexibility for that.

I did put / get / get_and_update and works perfectly.

Hope it scales and performs well for large sets.

5

Cachex is an amazing library. Every time I used it I’ve been very satisfied.

2 Likes