Ordering issue with erlang queue and an ETS table

Questions / Help
,
1

I have a system which I need to feed data into and maintain. Currently I have a GenServer that communicates with an ETS table and inserts items into the table. As part of this I have a FIFO erlang queue, when an item is added to the ETS table I am pushing it to the queue to be consumed from elsewhere.

For some reason the ordering of the items in the ETS and Queue are different and I cannot work out why. This is the code I am using to push the item into the ETS table and Queue.

with :ok <- Queue.add(job), true <- :ets.insert(table, {id, job}) do
    PubSub.broadcast!(BrokerEx.PubSub, @topic, {:added, job})
end

When I dump the contents of the table and the queue they are not the same. The Queue in this context is a simple GenServer that manipulates Erlangs built in queue module.

Where am I going wrong?

1 Like
2

I’ve worked it out. Using table = :ets.new("table", [:set, :protected]) creates an ETS table with the type of set and there is no guarantee of the order with this table type.

To be honest I think I need to use a RDBMS at this point anyway since I need to have some persistence for this data anyway.

1 Like
3

Yep, was about to say “why not use a database”? You can also use timeseries if needed – not so hard to install the timescale Postgres extension. Or you can just queue things up in Redis and have background workers pull them away from there and store them in a regular DB.

1 Like