Hello

• We’re using Oban.Pro 1.14
• we have 23 queues
• we use partitioning
• we use uniqueness
• we use local limits
• we use global limits
• we use rate limits
• we use DynamicCron heavily (1800+ cron entries)
• we have multiple nodes
• we process millions of jobs a day

We are having an issue where our Ecto.Repo ETS table is slowly growing until we get an OOM (currently reaching our limit of 8GB).

Our Repo is mainly used for Oban.

I am trying to dig deeper into this to try to isolate where this can come from, but it seems related to internals of Oban

Here we are using 105MB after a few hours:

image1922×174 18.3 KB

Some of the ETS table details at this point:

• has 16178 references to the oban_jobs table
• has 12 reference to name: "ecto_*

You can find a small portion of it at https://gist.githubusercontent.com/mathieurousseau/6e6529da36bc09927fb9227ed4acbebf/raw/72f46766b6acf9aede031d820e70df59577afd10/gistfile1.txt

Something I have isolated:

• The leader ecto ets table does not grow if it does not process any jobs (probably only taking care of main Oban business and the Cron)

What can I provide to help with this investigation?
Is there some obvious configuration that you know could mitigate this? (using prepare: :unamed on ecto configuration did not change anything)

Thanks.

Mathieu