3000 windows services on a single node, that can be stopped, started, paused and upgraded individually. So yes, looking to do the same with Elixir/OTP. The work does not have to be distributed, so all 3000 on a single node is no problem, the main requirement is the ability to stop, start pause and upgrade individually. We thought Umbrella is great because then all 3000 services can talk to the API application via elixir message passing, so a more succinct question would be how to have:
An API application on the same node and BEAM as 3000 gen servers that can be stopped, started, paused and upgraded individually. With Kubernetes/GRPC I think we would put each service in a docker container, this would be much heavier than elixir but it does fit the need to manage the services individually.
You might find this thread (and the blog post and course mentioned in it) of interest:
Dave doesn’t even use umbrella apps for them, preferring to just make each micro service a separate ‘normal’ Elixir app. I haven’t finished all the Elixir learning material I want to yet (got a few more books to go), but as of now I think I am going to be following Dave’s way of doing things as it just makes a lot of sense to me. If you haven’t got his course I highly recommend it.
With 3k services this would result in 3k nodes, which would mean roughly 9M node-interconnections, since the BEAM wants to interconnect all nodes with each other. This won’t work well. You need to replace the messaging by another mechanism that does not require the full mesh.
Also you’d load the BEAM 3k times, while by putting some if not all of those modules alltogether on a single bare machine without docker, you can still update the modules one by one…
With Elixir and advantage would be to have all services on the same node using message passing for intercommunication, but the question would be how to stop, start, pause and upgrade each service individually. Each service seems it would be a GenServer, but maybe an application?
Thanks for the link to dave’s article. I had read it, but after re-reading it after your mention a light-bulb might of gone off:
which (and I apologize for my newbness), I guess he is using filepaths in his mix.exs dependencies section. And worst case scenario is we could use a git-deploy type scenario where we can pull code down to the production node and then use a remote iex to reload the application and be able to update the deps/applications without affecting the other applications on that node.
I’d prefer a completely stand-alone private hex server that can run
internally where we can upload our internal packages to. I have a
half-implemented one for rebar3 which I am hoping to complete one day
I don’t like using version control being central to how I distribute my
dependencies. Then I must use a VCS which is supported, I must use
one repo per dependency. The language dependency management shouldn’t
care at all what VCS I am using and it should not make that decision for me
But until my private package server is actually done, we also uses git
repo dependencies for our private stuff.
This is the potential misunderstanding I wanted to address in my first reply. As far as I can tell a single node runs exactly one “primary application” - the other included applications act as libraries to the primary application by becoming part of the primary application’s supervision tree.
It’s a peculiarity in the OTP naming convention - a library application does not implement the Application callbacks and therefore cannot be started or stopped (as an application). So for example Poison is an “OTP application” but it’s a library application for use by a primary application.
$ mix new app_name
creates a library application - not a primary application. For a primary application
$ mix new app_name --sup
is required - that will include the application callback to create the supervision tree.
An umbrella project still contains just a single primary application - one of the “applications” implements the application callbacks and starts the supervision tree while the other (included) “applications” simply become part of the first application’s supervision tree.
but the question would be how to stop, start, pause and upgrade each service individually. Each service seems it would be a GenServer, but maybe an application?
In this situation the term service could cause some confusion/ambiguity. Handling a single source may require a number of processes, possibly even a small library application that could be designed to have it’s processes handling the source started, paused, and stopped. Upgrades could be trickier. In general hot code reloading targets code at the module level - and in many use cases the recommendation is to avoid using/supporting it because it tends to make the overall design much more challenging.
The work does not have to be distributed, so all 3000 on a single node is no problem.
The issue is
it’s not exactly clear how severe a workload handling a single source is
how capable the physical/virtual CPU is that the node resides on.
While a node will spread all its processes over all the cores of the CPU - it can’t scale by utilizing additional CPU’s (short of spawning processes on other nodes that reside on a different CPU). So while the total number of processes shouldn’t be a problem for a single node, the actual workload of handling 3000 concurrent sources could potentially be too much work for the CPU the node is executing on. If that is the case the solution design will have to account for the eventuality of distributing the workload over separate nodes each executing on their own CPU.
Taking service oriented design principles into account it may make sense to avoid sharing a node configuration database across multiple nodes but instead have a separate “configuration node” which supplies the other nodes with their configuration information when they start up (which could also route start, pause and stop requests to the appropriate node).
Ultimately the design details are affected by the expected workload and expected capability of your deployment platform.
We are building an API which talks to around 50 services which is not huge, but they are all different providers which understand SOAP, XML, JSON, Rest etc,. And, we have had good success with just putting them in different modules. I think just having 3000 genservers with supervisors may work out without a lot of complications. Like @NobbZ mentioned creating 3K nodes is definitely not a good idea. You will also have to tune your :hackney settings (if you are using HTTPoison) as it sets the max connection limit to a default value of 50.
I would personally do the simplest thing possible, by building out some common utilities which the 3000 services can use. And run them on a single node.
That is not entirely true, a single node can run as many applications as you want to, and each application can have its own supervision tree. The only limitation is that only one instance of each application (identified by name) can be started.
You can call :application.info()[:running] to inspect what applications are running on the node. For example, bare IEx shell outputs:
which means that logger, iex, elixir and kernel are the applications with the supervision tree (the PID in the list is the PID of application’s application master process - which is not the PID of the top supervisor). compiler and stdlib are library applications.
Are those 3000 “services” independent of each other?
Are they actually stateful?
Do they actually need to do background processing?
Integrations: This is where things go off the rails, we injest and transmit data, mostly in XML via HTTPS with over 3000 sources. An App per seems out of place, in most cases we have a REST or Soap client that listens to a queue and takes action, much like a background task. A lot of them will also spin up a GenServer per source (and this should be singleton across the entire cluster) and poll the source for data.
It seems to me that the REST/SOAP/XML is just “glue” to let the main service coordinate with those remote services?
If that is the case, you don’t even need to spin up 3000 GenServers ahead of time: Just make each service its own module, with plain functions, and “just call it” from wherever you need.
You will most likely need to add some abstractions on top if you want the call to be made non-blocking and whatnot. At the simplest level, spawn a process to do the calculation.
In those abstractions, you can easily spawn this process in a completely different machine, and the rest of the code won’t even know the difference.
So you could start with a beefy machine running just a single BEAM process (it will take over all the CPUs if you have a really beefy machine with multiple physical CPUs), and only if that appears to be not enough you can add a second beefy machine.
Regarding upgrading code, with OTP releases you can do hot code upgrades while the system is running with no downtime. I believe your use case is exactly what it was designed to do.
There’s quite a few details to work out, of course, mostly about the nature of your “integrations” and where do they get their data, state and whatnot.
Don’t worry too much about how you’d package it all up; in the end you’ll have an OTP release, one “primary” app that could just be a facade; the rest of the work will be done by the rest of the Applications (of which you can definitely have 3000, an application just needs to return a supervisor tree).
AFAIK, the only difference of the “primary” app is that if the BEAM can’t start it successfully, it will terminate the actual BEAM process entirely, as there’s no point in keeping the BEAM running if the primary app can’t start.
Running 3000 services in a single BEAM node should not be a problem. You basically start one (or more) processes per each service, and that’s it.
The second part of your requirement is indeed tricky. If you can afford to restart everything, your life will be much simpler. If not, then you must enter the realm of code reloading. Some basic instructions are available here.
In some simpler cases, this might work out of the box, If you actually have 3000 different modules (which I somehow doubt, but who knows ¯\(ツ)/¯ ), and cache previous release builds on the build server, then I think (not 100% sure though) that distillery will be able to detect the change and generate a correct appup automatically.
In more complicated cases, you might need to hand code an appup file. You can find some basic examples here. As far as I understand, appup is quite flexible. Among the low-level instructions there is apply which allows you to invoke a series of functions in an arbitrary order, so I it should be possible to do perform any kind of upgrade logic, no matter how complex it is.
I think that in a discussion like this it is important to stick with the official terminology in order to minimize confusion - so I think you meant to state:
So you could start with a beefy machine running just a single node (it will take over all the cores if you have a really beefy machine with multiple physical cores), and only if that appears to be not enough you can add a second beefy machine.
Because:
A BEAM process is scheduled to run on a single core by one of the node’s schedulers. A BEAM process can move to any scheduler within the node and therefore can run on any core of the CPU but at any one time is either executing or waiting on a single core of the CPU the node is executing on. A BEAM process cannot leave the node it’s executing on (sending a process function and state is more a matter of cloning).
It’s the “Bogdan/Björn’s Erlang Abstract Machine” (BEAM), the Erlang VM that runs on behalf of the node that has access to all the cores (not CPUs). By extension the node has access to all the cores of the CPU the node is executing on (WhatsApp was reportedly using CPUs with 10 cores). However the node is confined to the CPU that it is executing on - so on a true multi-processor (rather than multi-core) architecture the node cannot take over additional CPUs - the best it can do is spawn a process within another node that is running on another CPU (which could be on the same PCB or somewhere across a network connection).
As I stated in my first post I wasn’t entirely sure there was only “one application” - now the logger having its own supervision tree is suggestive of the intent to support “multiple ‘user’ supervision trees” (for the lack of a better term, :kernel, :elixir, etc. I would consider “infrastructure” supervision trees/applications) though I’m still foggy on what is considered “reasonable” practice.
The release file format supports multiple applications by necessity as the “infrastructure” applications have to be explicitly listed in addition to the “user” application. But there seems to be no direct constraint preventing having multiple “user” applications in the same release. But just because it’s possible doesn’t necessarily mean it’s a good idea to have multiple “user” applications in the same release - primarily because that could suggest a certain level of coupling - coupling that might be better served within the same supervision tree.
Two unrelated “user” applications could be in the same release for efficiency reasons - i.e. to share the infrastructure of the node. However it would seem more logical to have unrelated or loosely coupled applications in separate releases - unless a single node can only service one single release (which could make sense as two releases could specify different ERTS versions).
To me there seems to be a certain lack of clarity when it comes to the higher granularity concepts of the “Elixir/OTP alternative” to microservices. On an abstract level a microservice is simply a piece of software designed according to the principles of service orientation that operates within a deployment environment tailored to running and managing microservices. While a running instance of a microservice is typically constrained to a physical machine or specific instance of a virtualized environment that instance could appear on any one of the available physical machines or virtualized environments. The way microservices scale seems straight forward.
Meanwhile the discussion about the “Elixir/OTP alternative” seems to always revolve around processes, supervisors and usually a single supervision tree. However a single supervision tree seems to be practically confined to a single node and therefore CPU. In order to scale further it seems to become necessary to shift gears and start thinking about “OTP applications designed according to service oriented principles” and how to appropriately distribute responsibilities across any number of collaborating OTP applications. This raises questions that simply don’t come up when primarily thinking about single node (primary application) solutions:
Does it make sense (in production) to run multiple nodes on a single CPU or is it better to run all primary applications destined for the same CPU on a single node (provided the primary applications can use the same version ERTS)? What are the limitations and constraints?
Do all primary applications running on a single node have to be part of the same release or is it possible to have multiple releases (with distinct primary applications) for a single node? What are the limitations and drawbacks?
If they even might communicate then all within the same Node is better, less overhead, better scheduling and work distribution.
Same release, but that is what Umbrella’s are popular for.
Personally I package up near all my application as dependencies then just have a main ‘MyServerRelease’ project that just depends on them all and does nothing else, just for making releases. I’ve found it the easiest back to my Erlang days.
So far I’ve only come across umbrella projects where only one of the applications is the (top level) primary application, while the rest are merely library applications - i.e. the entire umbrella project is dedicated to assembling one single supervision tree.
Personally I package up near all my application as dependencies then just have a main ‘MyServerRelease’ project
But that sounds like it’s necessary to deploy the whole “ecosystem of applicatons in a big bang” rather than having the convenience of just deploying the one application that was actually changed - which is a typical “microservices expectation”.
Unless I’m misunderstanding your terminology, user applications can include “infrastructure” applications . It’s pretty standard to have multiple user applications running together on the same node.
Same within the same node is definitely simpler, and should be the first approach to consider.
Multiple nodes on the same machine is basically mainstream microservices (multiple OS processes), and it might make sense in some special situations. Two cases that come to mind:
Isolated deploy without the hassle of code upgrades. Split the system in two (or more) nodes, and then you can deploy/restart them separately, while other nodes are running.
Isolated failures in the case of a NIF or port driver. If you’re running in-process custom native code, then moving that code (and a minimal Elixir/Erlang code surrounding it) to a separate node ensures that e.g. a segfault in nif won’t take down the entire system.
In both cases I wouldn’t use distributed Erlang, but rather have nodes communicate via e.g. HTTP.
Newcomers to Elixir/OTP have a certain preconceived notion of what an application is - and it isn’t anything close to the way OTP uses the term which makes effective communication extremely difficult and is a setup for all sorts of misunderstandings and false expectations.
When I use the term “user application” I’m trying to zero in on an entity which implements the Application callbacks and therefore initiates the creation of it’s own supervisor tree - while at the same time not being part of the Erlang/Elixir runtime system. As has been mentioned :kernel, and :elixir are technically primary applications as they have their own independent supervisor tree but …
… if you talk to someone from JVM/CLR land they’ll agree that the runtime is essential for running of their application but they’ll give you a rather stupefied look when you start referring to the runtime as an “application”.
One of the worst offenders in this situation is the notion of a library application - how would people take it if we referred to React as an application? Yet it is the notion of the library application that gives people all sorts of initial expectations when it comes to umbrella projects - because it gives the impression of a workspace for developing multiple applications (each with it’s own supervision tree) when in fact it is usually just used to segregate the functionality that will ultimately run in one single supervisor tree (and therefore is confined to execute on one single node and therefore CPU - if I truly have multiple applications then I would expect that I can put each application on a separate node - which of course can’t be done with a library application).
It’s pretty standard to have multiple user applications running together on the same node.
Where? I’m not saying it can’t be done - clearly it can - but the typical scenario that seems to be presented is multiple applications (each with their own supervisor tree) that comprise the runtime system plus some optional environmental services (altogether comprising the “infrastructure”) simply to support the one, singleapplication (with it’s own singular supervisor tree) that we are actually interested in running.
I am confused. What does CPU mean in this context? A single node will create a scheduler for every core of every CPU, so a single node will happily take over all your cores and CPUs, and work will be done on all of them.
Joe Armstrong has said that Application was really a misnomer, and Component should have been used instead.
With that in mind, a library application just becomes a bunch of functions. You need to make it into an application in order to bundle them in a release, that’s all.
