nono.31
Oban - Caching Ecto Statements ETS Table memory leak -> OOM
Hello
- We’re using Oban.Pro 1.14
- we have 23 queues
- we use partitioning
- we use uniqueness
- we use local limits
- we use global limits
- we use rate limits
- we use DynamicCron heavily (1800+ cron entries)
- we have multiple nodes
- we process millions of jobs a day
We are having an issue where our Ecto.Repo ETS table is slowly growing until we get an OOM (currently reaching our limit of 8GB).
Our Repo is mainly used for Oban.
I am trying to dig deeper into this to try to isolate where this can come from, but it seems related to internals of Oban 
Here we are using 105MB after a few hours:
Some of the ETS table details at this point:
- has 16178 references to the
oban_jobstable - has 12 reference to
name: "ecto_*
You can find a small portion of it at https://gist.githubusercontent.com/mathieurousseau/6e6529da36bc09927fb9227ed4acbebf/raw/72f46766b6acf9aede031d820e70df59577afd10/gistfile1.txt
Something I have isolated:
- The leader ecto ets table does not grow if it does not process any jobs (probably only taking care of main Oban business and the Cron)
What can I provide to help with this investigation?
Is there some obvious configuration that you know could mitigate this? (using prepare: :unamed on ecto configuration did not change anything)
Thanks.
Mathieu
Trending in Questions
Other Trending Topics
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
Courses that'll move you from confusion to "Aha, now I get it!"
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 10 of 23 Posts
benwilson512
What version of Ecto are you using? I seem to recall some isssue in Ecto related to this that was fixed at one point.
dimitarvp
Probably talking about this?
sorentwo
As indicated, this is fundamentally an Ecto issue. There are improvements in more recent versions of Pro to compensate for this issue by disabling prepared statements for highly dynamic queries.
Aside from upgrading Ecto and Pro, you could disable prepared statements for all Repo queries with
prepare: :unnamed(double check the options in the Ecto docs).nono.31
About the Ecto fix post:
You can find me saying on that post that last version of Ecto did not solve the issue for us
We tested 1.5 RC but issue remained. We are on latest Ecto and
prepare: :unnameddid not workI will try to create a dummy project to reproduce this behaviour.
dimitarvp
Whoops, sorry. No clue then.
sorentwo
That should disable prepared statement caching entirely. Are you certain you configured that for the Repo that Oban is using?
Please do. If you have a reproduction I’ll gladly investigate.
nono.31
yeah I am sure I have configured
prepare: :unnamedon the RepoThe catch though, I have doubled check on our deploy and:
prepare: :unnameddone inruntime.exsdoes not workI will put it in the
config.exsand test again.nono.31
Well, that did not make any difference. I doubled check the conf was loaded and ETS table still grows
sorentwo
That sounds like an issue with
ectoordbconnectionitself then. Oban makes judicious use ofprepare: :unnamedfor frequent, variable queries, and it would be an problem if the option isn’t being respected.nono.31
yes, that’s probably it.
I have a dummy project that I need to clean up that I will share shortly.
But here are the queries that are still being cached:
all,delete_all,update_allExample of the largest one (141k):