Oban - Caching Ecto Statements ETS Table memory leak -> OOM

[sorentwo]

@Schultzer

A simple solution to avoid this in the future would be to use a library that does not cache queries in ETS at all

That would avoid part of the problem, but not all of it. There’s a fundamental issue with that query structure because there’s a hard limit on the number of unions (and it doesn’t scale very well). That’s also entirely fixed with v1.6, it’s not just a fix for the caching side of things.

There’s also a benefit to using prepared queries, which require some element of caching to know what’s prepared.

[Schultzer]

True, I was referring to the ETS issue thats why my library is doing all that without ETS.

[ataideal]

Thank you @Schultzer and @sorentwo for the replies.
Just a small correction on what I said previously, one of the arguments is a list of maps, not list of strings (that might complicated the queries even more).
For now we will disable the oban ratelimit and implement some sort of hash + redis solution, but as soon as we update to oban v1.6+ we will double check the improvements on this again.

[sorentwo]

@ataideal

Just a small correction on what I said previously, one of the arguments is a list of maps, not list of strings (that might complicated the queries even more).

Great. That doesn’t matter at all for the queries in v1.6