matteosister
Graceful shutdown of Elixir app in docker
In our company, we have an application that runs rabbitmq consumer in elixir on AWS ECS. So everything run on docker containers, and we scale horizontally by adding more containers.
For this reason many node go up and down during the day. As the business grow (and the number of messages handled by consumers) we are facing problems with workers interrupted in the middle of the message handling. We are trying to do whatever we can to mitigate the problem by implement transaction that commit at the end of the job, but we are also trying to understand if it’s possible to watch for SIGTERM message and clean up before shutting down.
I found this post on stack overflow that basically tells that erlang/elixir is not able to handle unix signal.
Which approach you suggest in a case like this? Anyone cares to share his/her experience? Many thanks in advance!
Marked As Solved
josevalim
Erlang ships with two amazing command line utilities which you can use to run any application and connect to it any time you want. They are called run_erl and to_erl:
run_erl ./my_app /dir/for/logging iex -S mix
The command above will execute iex -S mix and give it a name of my_app and will log any entries to “/dir/for/logging”. Make sure the logging directory exists otherwise run_erl may fail silently.
Now you can connect to the iex terminal of that node at any time by doing this:
to_erl ./my_app
This means that, if you use run_erl to start Elixir with IEx inside Docker, you can connect to IEx and issue :init.stop/0 for proper node shutdown. :init.stop/0 will go application by application and shutdown their supervision tree respecting the configured timeouts. You can automate it by running:
echo ":init.stop" > to_erl ./my_app
And that’s it! If you are using releases, for example via Distillery, they handle this stuff automatically for you.
Also Liked
josevalim
You could have a plug that checks for a given application key and returns true if you should return 503.
Then you could have a function in your application, let’s say in the MyApp module that does this:
def slow_shutdown do
# This is the value that the plug will check to know if it should return 503
Application.put_env(:my_app, :shutting_down, true)
# Let's wait a second before finally shutting down
Process.sleep(60_000)
# Finally turn everything off
:init.stop
end
Now you can use the same tips I used above and instead of echo ":init.stop" > to_erl, you can do echo "MyApp.slow_shutdown" > to_erl. If you are using distillery, it is likely distillery ships with an option to run a module+function in the currently running node.
svsdehh
thank you very much for your tip.
we will now see, that/how we can execute the code in the current node. that should do it then.
svsdehh
we have a very similar but slightly different use case.
we want to deliver a 503 on a health endpoint for some seconds on a sigterm, before the server is shutting down. we are using distillery for a phoenix micro service and the service is running in docker.
the reason for the 503 on the health endpoint is, so that the HAProxy in front of the service is recognizing that the service is going down before it is really down. so the service wont get any more request from the haproxy.
has anyone solved a similar situation or has any suggestions, how to solve this?
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance








