Fake51
Readiness/liveness issues
I have a catch-22 issue that I’m struggling somewhat with and could use some input/experiences from others.
I’m developing a stocklevel tracking service that uses mnesia and which gets deployed to a kubernetes cluster. My issue is that new nodes starting up will not get connected to the already connected nodes if I use the kubernetes readiness property - however, as part of starting up I want a node to get a copy of the mnesia tables, ram copy only. I’m not marking it ready until it has connected to the clustered nodes and received it’s copy.
Does anyone have experience using either libcluster or peerage (which is what I’m using currently) and connecting nodes before kubernetes sees the new node as ready?
Marked As Solved
Fake51
I found a solution to my issue, it seems. it’s documented but poorly.
My setup is using a headless service in kubernetes - that handles the DNS, so the pods get IPs of each other by looking up their own hostnames. That’s bog standard and easy. However, the headless service doesn’t publish the IP of pods that don’t show as ready, unless you specifically add publishNotReadyAddresses to the service config. Set that to true and it works.
Thanks for the help, benwilson512!
Also Liked
benwilson512
Hey @Fake51, here’s what I do.
Here is my health controller:
defmodule Sensetra.Web.HealthController do
use Phoenix.Controller, log: false
def alive(conn, _params) do
json(conn, %{alive: true})
end
def ready(conn, _params) do
if Application.get_env(:sensetra, :ready) do
json(conn, %{ready: true})
else
send_resp(conn, 503, "")
end
end
end
Notice how the alive endpoint always returns true, but the ready endpoint is conditional on an application environment setting. In my config, that value defaults to false. Then, I just have a tiny genserver as the very last entry in my supervision tree that sets the value to true. This ensures that the application doesn’t indicate that it is ready until the entire supervision tree has booted.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #javascript
- #code-sync
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance








