cjbottaro

cjbottaro

Supervisors in a container/pod world

The product I work on leans heavily on Kubernetes. We typically expect OS processes to crash and cause pods to be restarted and that hooks into all sorts of monitoring and alerting.

Our Elixir app starts a lot of supervised processes. If one of those processes crashes repeatedly, the supervisor just gives up on trying to restart it, but the OS process stays up. Thus Kubernetes won’t restart the pod.

I know there is a way to tell a supervisor to try to restart the process indefinitely, but is there a way to say “if this process can’t be restarted, exit the entire OS process?”

Or is there a better way to think about this problem? OTP is great for a lot of things, but I feel like modern container orchestration frameworks kinda supersede some features.

Thanks for the help!

Most Liked

LostKobrakai

LostKobrakai

You could tinker with :max_restarts/:max_seconds on supervisors, but depending on the tree depth that might not be the most efficient way to go about this. You could also put a process elsewhere in the system, which monitors our “critical” process and if it receives an exit message does System.stop(…).

dimitarvp

dimitarvp

Looks like that it’s maybe time to switch to a NIF-based JSON parser and/or use a streaming one? Options:

IMO when you get to the point of a worker timing out due to having to process huge input, a streaming parser is your only viable option. But maybe jiffy or jsonrs would do the job because they should parse several times faster.

I am not sure OTP can help you a lot here; or if you really want that and don’t want to switch the JSON parsing library then maybe extract out the JSON parsing code into a separate app.

LostKobrakai

LostKobrakai

While there is a application supervision tree, it likely works differently than you expect. The pid returned from Application.start/2 callback is never restarted. When it crashes the application is considered failed and stopped. Restarts essentially only happen to children of supervisors you control – there’s no implicit supervisor restarting things at the application level.

So you could probably do a setup like this:

        ┌───────────┐
        │  RootSup  │ Allow no restarts
        └───────────┘
              │
      ┌───────┴──────┐
      │              │
      ▼              ▼
┌ ─ ─ ─ ─ ─ ┐  ┌───────────┐
   ImpProc     │  ApplSup  │ Default restart limits
└ ─ ─ ─ ─ ─ ┘  └───────────┘
                     │
               ┌─────┴────────┐
               │              │
               ▼              ▼
         ┌ ─ ─ ─ ─ ─    ┌ ─ ─ ─ ─ ─
             Rest   │       Rest   │
         └ ─ ─ ─ ─ ─    └ ─ ─ ─ ─ ─

There’s also a :significant setting for OTP, which I’m not sure if you can use it yet with elixir:

Where Next?

Popular in Questions Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
Kurisu
For example for a current url like http://localhost:4000/cosmetic/products?_utf8=✓&query=perfume&page=2, I would like to get: ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New

Other popular topics Top

9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43806 214
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31307 143
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New

We're in Beta

About us Mission Statement