Harrisonl
Deploying Elixir into ECS causing many "'global' at node :"xxxxx@10.0.X.X" requested disconnect from node :"xxxx@10.0.X.X" in order to prevent overlapping partitions"
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service.
Currently when I deploy, new tasks will attempt to join the old cluster as they a being brought up (potential red flag). I think (but not sure) that this is causing the above message to happen a lot on deploy, as once new tasks reach a healthy state, old tasks drop off causing incomplete partitions as some tasks may be notified first.
This is causing many issues, especially with Horde as it tries to connect to the old nodes (something between Libcluster and Horde not communicating correctly) resulting in every deploy having our Horde registry be reset and orphaned processes.
I’m not sure if this is actually the issue though or not or if there is something else I’m missing. I’m using LibCluster DNSPoller (1 second interval) for node membership and OTP 26.
My questions are:
- Is this expected for this setup (dynamic cluster with nodes joining / leaving frequently) or am I missing something?
- I couldn’t find a concrete answer on this, but is it better to have the new nodes not join the old cluster when they are brought up (e.g. ensure they all have the same version etc.)?
Most Liked
gfviegas
Had the exactly same issue deploying in a Fargate ECS cluster with 2 tasks (each with one container with a phoenix app using libcluster dns poller).
Besides desabling the prevent_overlapping_partitions as recommended by @schneebyte I’ve also decreased the Min running tasks % in my ECS Service to 50% to ensure a task will still be up to converge a replicated cache in the newly created nodes.
It fixed for my use case.
To give a little bit of ‘how to’ for newbies (like me) in clustering in elixir, if you are using releases and wish to disable the prevent overlapping partitions option, you should add a new line in rel/remote.vm.args.eex (or similar) with the followng content:
-kernel prevent_overlapping_partitions false
If you are not using releases and need a quick fix, you can also set an environment variable ELIXIR_ERL_OPTIONS with this setting, such as ELIXIR_ERL_OPTIONS='-kernel prevent_overlapping_partitions false'.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"