Memory management - is there a general rule?

Maybe you should have used ETS table instead of list? If there are random updates it can make it slower even further.

If the order of IDs in you list doesn’t matter, you can keep the list sorted to get better performance when inserting or removing IDs.

defmodule IDList do
  def new do
    []
  end

  def add(list, [id | ids]) do
    add(add(list, id), ids)
  end

  def add(list, []) do
    list
  end

  def add(list, id) do
    insert_unique(list, id)
  end

  def rm(list, [id | ids]) do
    rm(rm(list, id), ids)
  end

  def rm(list, []) do
    list
  end

  def rm(list, id) do
    remove_unique(list, id)
  end

  def insert_unique([], id) do
    [id]
  end

  def insert_unique([top | rest], id) when id > top do
    [top | insert_unique(rest, id)]
  end

  def insert_unique([id | rest], id) do
    [id | rest]
  end

  def insert_unique([top | rest], id) do
    [id, top | rest]
  end

  def remove_unique([], _) do
    []
  end

  def remove_unique([top | rest], id) when top < id do
    [top | remove_unique(rest, id)]
  end

  def remove_unique([id | rest], id) do
    rest
  end

  def remove_unique(rest, _) do
    rest
  end
end

ExUnit.start()

defmodule IDListTest do
  use ExUnit.Case

  test "inserting" do
    list = IDList.add(IDList.new(), [1, 1, 2, 1, 2, 2, 1])

    assert [1, 2] = list

    assert list == IDList.add(list, [])

    assert [1, 2, 3] = IDList.add(list, 3)
    assert [1, 2, 3] = IDList.add(list, [3])
  end

  test "removing" do
    list = IDList.add(IDList.new(), [4, 3, 2, 1])
    assert list == IDList.rm(list, [])
    assert [1, 2, 3] = IDList.rm(list, 4)
    assert [1, 2, 3] = IDList.rm(list, [4])
    assert [1, 2, 3] = IDList.rm(list, [4, 4])
    assert [2, 3] = IDList.rm(list, [1, 4])
    assert [2, 3] = IDList.rm(list, [4, 1])
  end
end

Or using a MapSet could be better too. But in any case 1000 ids “a few times per second” is small and you should not notice a GC pause for that.

As @hauleth notes, :ets may be the right call here. It has the further perk of not creating garbage collector pressure, since its items are stored in its own managed memory space instead of within the process memory.

When it get slow. However, as other has noted, what you are doing should not be slow; and even if it does get slow, there is other ways to make it faster, such as ets.

I am sure I am have a GC pause for something related to this list. 1) Because I could tweek it a little bit by changing the GC parameters of the GenServer. 2) Because the separation of the list into another process made everything smooth.

But you are right its not the list itself. to be a bit more precise its a list of unordered integers that provide available unique ids (within a specified range).

Am I right that a list updated with [head|tail] triggers little memory movement (sorry i don’t know the right term)?

Does updating a list within a map create significantly more “memory movement”, so that the GC has to work?

%{a: mylist, b: someotherstuff}

For example does a change in a implies that b has to be copied?

No, the value associated to b should not be copied if you change a. Rather, a new map that references the same b should be created. The details probably depend on the size of the map (I guess bigger maps are implemented as HAMT, smaller ones possibly not), but in any case b should not be copied if something else in the map changes: the implementation will generally try to reference as much as possible of the old immutable data structure.

I suspect there is something more to your case, could you maybe share some code?

I ran a quick benchmark based on a list of 1000 ids. When adding 500 existing ids and 500 new ids, then removing 500 existing and 500 not existing, ETS was the fastest.
Adding and removing 200 IDs, MapSet was faster (still on a 1000 IDs list at start).
My quick and dirty unique list implementation was the slowest in any case.

But ETS comes with its own new problems because it is like a mutable array. Now, that should not be a problem since you use another process currently, so you already have a “kind of mutable” state. But if you need to do a lot of things locally with those IDs, don’t use ETS if it is impractical I guess.

But I agree, show us some code!

ETS is a nice solution to your problem. One consideration, most guides show an ETS table “managed” by a GenServer but this is suboptimal. ETS allows concurrent access and a GenServer in front of it limits the concurrency. Just create a table, maybe in your Application, and access it through it’s name from a module containing only functions.

I think you’re confusing the practice of having a specific process that owns the :ets table with the practice of serializing access to that table through the genserver. The two are orthogonal. You can setup a named table with read / write concurrency that is accessed directly, but still is owned by a supervised process. It is generally a best practice to put :ets tables under such managed processes because it allows them to be started / stopped / restarted effectively within your application’s supervision tree.

It should probably be said that an entire book chapter could be dedicated to these situations (I believe there are a few in existence!). I’d maybe go as far as to say it’s one of the main quirks with immutable data, since you can find yourself copying the data over and over instead of what is a simple pointer update in mutable languages.

Also, a thousand IDs should not cause what you are describing; which makes me think you are replicating the data. If the ETS option doesn’t solve your problem, I’ve picked up a few tricks when dealing with this type of scenario; which might be applicable for you:

1. you can inspect the process’ memory usage using Observer. Runaway increases in memory may be a sign that old data is not being cleaned up as you update the ID list.

2. if you’re processing the IDs each in a dedicated process, you can “monitor” the process, listen for the :DOWN message, and then force garbage collection (:erlang.garbage_collect()). Not ideal, but it can work. You can target particular processes as well (eg. parent process).

3. state (memory) can come along for the ride when using Task.Supervisor with anonymous functions. Here’s a nice explanation (not a bug). Short answer, MFA is preferred. If you’re doing any supervised processing in a loop, you may be inadvertently passing a lot of data around (locking it up).