cjbottaro
Supervisors in a container/pod world
The product I work on leans heavily on Kubernetes. We typically expect OS processes to crash and cause pods to be restarted and that hooks into all sorts of monitoring and alerting.
Our Elixir app starts a lot of supervised processes. If one of those processes crashes repeatedly, the supervisor just gives up on trying to restart it, but the OS process stays up. Thus Kubernetes won’t restart the pod.
I know there is a way to tell a supervisor to try to restart the process indefinitely, but is there a way to say “if this process can’t be restarted, exit the entire OS process?”
Or is there a better way to think about this problem? OTP is great for a lot of things, but I feel like modern container orchestration frameworks kinda supersede some features.
Thanks for the help!
Most Liked
LostKobrakai
You could tinker with :max_restarts/:max_seconds on supervisors, but depending on the tree depth that might not be the most efficient way to go about this. You could also put a process elsewhere in the system, which monitors our “critical” process and if it receives an exit message does System.stop(…).
dimitarvp
Looks like that it’s maybe time to switch to a NIF-based JSON parser and/or use a streaming one? Options:
- GitHub - rusterlium/juicy: A fast JSON parser for Elixir with some extra features · GitHub (last updated 5 years ago though)
- GitHub - talentdeficit/jsx: an erlang application for consuming, producing and manipulating json. inspired by yajl · GitHub (last updated ~11 months ago, Erlang)
- GitHub - kbrw/json_stream: Small but useful wrapper above erlang `jsx` to stream json elements from an Elixir binary stream. · GitHub (based on the above
jsxbut last updated 2 years ago)
IMO when you get to the point of a worker timing out due to having to process huge input, a streaming parser is your only viable option. But maybe jiffy or jsonrs would do the job because they should parse several times faster.
I am not sure OTP can help you a lot here; or if you really want that and don’t want to switch the JSON parsing library then maybe extract out the JSON parsing code into a separate app.
LostKobrakai
While there is a application supervision tree, it likely works differently than you expect. The pid returned from Application.start/2 callback is never restarted. When it crashes the application is considered failed and stopped. Restarts essentially only happen to children of supervisors you control – there’s no implicit supervisor restarting things at the application level.
So you could probably do a setup like this:
┌───────────┐
│ RootSup │ Allow no restarts
└───────────┘
│
┌───────┴──────┐
│ │
▼ ▼
┌ ─ ─ ─ ─ ─ ┐ ┌───────────┐
ImpProc │ ApplSup │ Default restart limits
└ ─ ─ ─ ─ ─ ┘ └───────────┘
│
┌─────┴────────┐
│ │
▼ ▼
┌ ─ ─ ─ ─ ─ ┌ ─ ─ ─ ─ ─
Rest │ Rest │
└ ─ ─ ─ ─ ─ └ ─ ─ ─ ─ ─
There’s also a :significant setting for OTP, which I’m not sure if you can use it yet with elixir:
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









