Will-W

Will-W

We’re using clustering but probably not in a way that you’re interested in. My current project is an embedded system of between 3 and 30 nodes which are all telecomms related equipment distributed around a site and networked together. The software on each node is responsible for configuring the attached hardware, but the nodes communicate with each other to co-ordinate this.

We do very basic cluster formation (manually, although I’m looking at libcluster) and then use :rpc.call() for the comms. This required very little faff to set up, and although we’re only in the early stages of development it seems to work well for us.

3
Post #11
onkara

onkara

@benwilson512 Are your apps containerized? Are you running on bare metal? What are you using for clustering?

0
Post #12
onkara

onkara

@egze Were you experiencing issues using Elixir libraries to handle periodical tasks? If yes, would you please share?

0
Post #13
benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

They are containerized, we use libcluster’s K8s service plugin, it works great!

0
Post #14
egze

egze

No, not at all. But we decided to use existing infrastructure tools that our DevOps support.

It also allows us to start periodical tasks with more memory, than what is needed to run the web application.

0
Post #15
chulkilee

chulkilee

We have a few services in Elixir

  • All run on docker
  • All run on k8s - except one (running on vm + docker + systemd - but we will move this to k8s too :wink: )
  • We don’t use hot code reloading at all - just using k8s way.
  • Some services are using erlang cluster, while some are just independent containers.

For services using erlang cluster:

  • We use libcluster to form a cluster from k8s dns, and it works very well
    • A while ago, I tried to set up erlang clusters on vms across AWS EC2 instances with ansible, and it wasn’t simple.. I don’t remember the details but probably due to epmd.
  • We need erlang clusters for global knowledge (e.g. websocket) and message passing between processes across nodes with minimal latency.. and distributed erlang is a great feat.
  • Some application uses Horde for global registry, and we have an issue on network split (link) - but other than that it works well as it is designed.

For services not using erlang cluster:

  • To distribute async jobs - we use oban. This is our preferred choice if a service already requires postgresql. Having persistent, auditable records are often “required”.
    • oban also have “unique” job and cron job features - so we don’t need to build global lock or global scheduler which is very nice.

On docker:

  • We have our own base image (compiling erlang, reusing precompiled elixir). Some codes are available at here
  • We haven’t hit any issues from erlang/elixir on docker.

On k8s:

  • k8s is not easy - lots of things to learn, CI/CD and local dev ecosystem is still evolving.. but it does make sense for certain circumstance :slight_smile:

Please keep in mind that you don’t need erlang cluster for scaling. Running independent instances behind load balancer is totally fine.

You WILL NEED erlang cluster when you choose to use any features only available in erlang cluster for scaling. For example, I’ve seen a presentation to add global cache on the top of database to reduce the latency.

6
Post #16
— All posts loaded —