cjbottaro

cjbottaro

Supervisors in a container/pod world

The product I work on leans heavily on Kubernetes. We typically expect OS processes to crash and cause pods to be restarted and that hooks into all sorts of monitoring and alerting.

Our Elixir app starts a lot of supervised processes. If one of those processes crashes repeatedly, the supervisor just gives up on trying to restart it, but the OS process stays up. Thus Kubernetes won’t restart the pod.

I know there is a way to tell a supervisor to try to restart the process indefinitely, but is there a way to say “if this process can’t be restarted, exit the entire OS process?”

Or is there a better way to think about this problem? OTP is great for a lot of things, but I feel like modern container orchestration frameworks kinda supersede some features.

Thanks for the help!

Most Liked

LostKobrakai

LostKobrakai

You could tinker with :max_restarts/:max_seconds on supervisors, but depending on the tree depth that might not be the most efficient way to go about this. You could also put a process elsewhere in the system, which monitors our “critical” process and if it receives an exit message does System.stop(…).

dimitarvp

dimitarvp

Looks like that it’s maybe time to switch to a NIF-based JSON parser and/or use a streaming one? Options:

IMO when you get to the point of a worker timing out due to having to process huge input, a streaming parser is your only viable option. But maybe jiffy or jsonrs would do the job because they should parse several times faster.

I am not sure OTP can help you a lot here; or if you really want that and don’t want to switch the JSON parsing library then maybe extract out the JSON parsing code into a separate app.

LostKobrakai

LostKobrakai

While there is a application supervision tree, it likely works differently than you expect. The pid returned from Application.start/2 callback is never restarted. When it crashes the application is considered failed and stopped. Restarts essentially only happen to children of supervisors you control – there’s no implicit supervisor restarting things at the application level.

So you could probably do a setup like this:

        ┌───────────┐
        │  RootSup  │ Allow no restarts
        └───────────┘
              │
      ┌───────┴──────┐
      │              │
      ▼              ▼
┌ ─ ─ ─ ─ ─ ┐  ┌───────────┐
   ImpProc     │  ApplSup  │ Default restart limits
└ ─ ─ ─ ─ ─ ┘  └───────────┘
                     │
               ┌─────┴────────┐
               │              │
               ▼              ▼
         ┌ ─ ─ ─ ─ ─    ┌ ─ ─ ─ ─ ─
             Rest   │       Rest   │
         └ ─ ─ ─ ─ ─    └ ─ ─ ─ ─ ─

There’s also a :significant setting for OTP, which I’m not sure if you can use it yet with elixir:

Where Next?

Popular in Questions Top

chokchit
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2733ms. You can configure how long re...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
Kurisu
For example for a current url like http://localhost:4000/cosmetic/products?_utf8=✓&query=perfume&page=2, I would like to get: ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
JulienCorb
I am trying to implement my new.html.eex file to create new posts on my website. new.html.eex: <h1>Create Post</h1> <%= ...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New

Other popular topics Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement