Term container supporting `push(term)` and `get_last_100()`

Questions / Help
1

Is there an easy to use ordered data structure that supports a push operation to add some term and an operation to fetch the last 100 (or n) pushed terms such that old values are discarded (a bit in the spirit of a cyclic buffer)?

I would like to have light-weight per-process RAM logs which only save the latest entries such that I can write a phoenix endpoint and introspect if latest log entries look normal.
Do you have any suggestions? Thank you.

2

The exercism exercises for a cyclic buffer use an okasagi queue under the hood, and also store the current queue size in an extra counter to be able to drop excess values without expensive counting

2 Likes
3

Thank you, I found this library CircularBuffer — CircularBuffer v0.4.1, I think I can use it for my problem.

4

Erlang’s built-in :queue should work fine for this, I’d think. You can of course use a library, but I’d likely implement this myself to have fewer deps.

To be clear, do you only ever want to save the last N entries, i.e. the oldest entry is dropped when N+1 entry is pushed? If so, you could use a {queue, current_len, max_len} tuple and a push/2 and pop_n/2 that implements your requirements.

Edit: sketching this out

def new(max_len) do
  {:queue.new(), 0, max_len}
end

def push({q, max, max}, element) do
  # at max len so drop element at head of queue
  q = :queue.drop(q)
  {:queue.in(element, q), max, max}
end

def push({q, len, max}, element) do
  {:queue.in(element, q), len + 1, max}
end

def pop_n({q, len, max}, n) when n >= len do
  {:queue.to_list(q), new(max)}
end

def pop_n({q, len, max}, n) do
  # exercise for reader ;)
  # could do recursive pop or split
  # https://www.erlang.org/doc/man/queue.html#split-2
end
2 Likes
5

To be clear, do you only ever want to save the last N entries, i.e. the oldest entry is dropped when N+1 entry is pushed?

Actually I don’t care if old entries are dropped immediately or in chunks when the size hits an additional larger threshold than N. I just want to always have at least N entries available for retrieval and don’t want it to use infinite amount of memory so some sort of dropping old entries is necessary.

This is elegant with the pattern matching def push({q, max, max}, element) do, nice. I can make something out of this sketch. Thank you.