Node discovery and connecting many clients to specific application nodes

We are reviewing our application architecture and wonder if there are recommendations or guidance on how we should connect the distributed nodes.

We have a lot of client applications which connect to a separate node to write content. This writer is registered as a global process which all of the clients send messages to. We have our own node discovery at the moment in the writer to connect to the client nodes.

In the future we plan to have one or two other specific nodes for other purposes, such as data APIs or monitoring. These are likely to be singular nodes as it is not a concern if they are unavailable.

We are using separate EC2 instances for reasons and can’t change this. We also have autoscaling meaning the IPs are not fixed or known beforehand. They can change at anytime when new nodes appear. Even the specific applications such as the writer get rotated periodically.

Are there any recommendations on how we should connect the nodes, either the clients to the specific applications, or which should form the cluster or manage the connection? We are ideally looking for something already maintained and not rolling out our own solution - unless our setup is unique which seems unlikely.

I have looked at a few libraries such as libcluster but not sure if that is the most suitable approach for connecting to a specific application. I have also seen a few which offer routing to nodes but they don’t seem to create the connection. I have seen a lot of examples where the same application is replicated and allows work to be distributed or for a failover approach. I couldn’t find much for specific applications having many clients.

Any guidance and recommendations on this is appreciated.