I am using the bridge network mode, with the custom peerage provider for srv records.
NAME RESULT OF ATTEMPT
api@172.31.13.73 true
api@172.31.6.233 false
LIVE NODES
api@172.31.13.73 (self)
Not really sure why it won’t connect. Any help is appreciated.
I struggled with this for a week and eventually got it working. The information in this thread is enough to point in the right direction, but there is a big gap between the right direction and a successful deployment.
I’ve written up a fresh guide on deploying distributed Elixir to ECS Fargate tasks with Service Discovery and libcluster’s DNS Polling strategy. Hope it helps save someone some time!
Thanks for this write-up. However, note that Route53 multi-value DNS will only ever return 8 records maximum. This limits the number of nodes in a cluster using this strategy to 8. Any more than that, and the results returned by the dns query will vary each time, causing libcluster to think nodes are coming and going, and you will see lots of disconnects/reconnects.
There is also a possible issue with the task IPs getting removed from the private namespace zone only after a task has transitioned to the STOPPED state (which seems like a major design flaw), not giving a task being shut down a chance to drain any pending requests and account for the A record TTL. I have not fully investigated this, however.
So after some investigation, it does look like the A records get removed immediately with the stop requests. It also appears that the application does not get a SIGTERM for quite a while after that – the container remains active all the way through to the end of the DEPROVISIONING state.
The limitation with the number of A records being limited to 8 stands, however. More than that and the cluster will experience lots of random disconnects and reconnects. It is possible that things may eventually stabilze (seems like it should in theory) – I am still looking at this.
