Correct if me I’m wrong, as best I can tell there aren’t any reasons to use mix run --no-halt in production vs releases. The marginal value always outweighs the marginal cost.
Marginal cost:
I’ve seen a lot of folks think that using releases means you MUST have a separate build vs application server/image/whatever. Heroku is a good example. You have a buildpack that contains Elixir and you compile your app with mix. What is the marginal cost of using releases on heroku?
Just one extra command! mix release, and then you run ./bin/myapp foreground instead of mix run --no-halt
Marginal value:
The marginal cost is low, what value do we get for that cost? There are some handy defaults surrounding remote console capability and heart (although frankly I’m a bit unclear about in what scenarios it’s going to act).
The PRIMARY value though is that releases will eagerly load all of your application and dependency code, whereas mix does so lazily.
For projects with many dependencies, there can be a massive latency spike when you first get requests / jobs that occur after starting your app with mix while it loads all your code. This can cause timeouts and error cascades. Sure the supervisor trees generally restart but there are situations where you exceed supervisor restarts per second limits and you end up having enormous portions of your app restart.
I agree. I believe that OTP releases for systems (aka services that need to run continuously), and escripts for CLI tools are the way to go. Maybe there are some special cases, but can’t think of any.
I wouldn’t say that eager loading is the primary value, although it depends on the use case. I suspect that many systems are not that highly loaded for that to make a difference.
I do think that the defaults you mention, which allows remote logging and start the system as the node (so you can remote observe it) are more important.
But above all, I like the self-contained, isolated nature of the release. You won’t pickup some unexpected module from somewhere else (e.g. mix app is not included in OTP release, so less chances of invoking Mix.env at runtime), and you can easily run multiple different systems (services) with different versions of Elixir/Erlang.
I also agree that the cost is marginal. But even if it takes longer to properly setup your own build and deployment, it’s a one-off cost. Pay the price once, reap the benefits many times
I’ve not used releases by now, but from what I’ve read about it in slack or otherwise online it seems to involve a lot of quirks and setup to get it to work properly. Installing elixir and using mix can be done by copy pasting a few (well documented) lines of code in your ssh session and the app is running. Comparing that to potentially fiddling around with building the release, dealing with any issues coming up and having to find solutions to those issues it doesn’t sound so marginal. Especially thinking of someone just coming into the language.
I still feel like releases are the better option long term and if releases are working for someone there’s probably no reason at all for mix run, but for me – coming from the upload-file-and-done php world – it feels nice to also have the option to skip all the ceremony. Even if it might be just for the quick and simple success feeling for novices in elixir.
How does the Elixir release process compare to the erlang release process? I recall while reading Learn you some Erlang for great good that the general vibe of putting together a release was “this is exceptionally complex and you should only do this if you really really need the hotloading capabilities”. I think this picture summed it up nicely.
I’m still a bit of a novice to this side of elixir and erlang, so can anyone speak to how the release process compares between these two languages?
The vibe you’re recalling from LYSE if I recall correctly has more to do with appups/hot upgrades - they should be generally avoided unless you really need the functionality and are prepared to deal with the complexity. Releases are very easy to build and use, and I can’t imagine not using them for deployment. Erlang suffered from poor tooling on that front for a long time, but that’s not been the case for probably 5 years or more.
Replying to both of these sentiments, I feel like a lot of the weirdness has been ironed out. Distillery is really easy to use at this point. Even the whole environment variables thing is nice and simple with the support of config :foo, "%{SOME_ENV_VAR}".
I have used releases for one production environment now; a webshop that does not need to be hot-reloaded, but I set up everything so that it is, to ensure that I get a feeling for how it goes.
I agree that using a release seems to be better than the mix run --no-halt. Until now I’ve created every release on my local machine, which meant that moving it to the external server was always somewhat of an issue; a release is a quite large binary file, and as every release is put in its own folder, tools like rsync cannot leverage speedups as you need to move the whole file instead of a diff. So building locally also has a very clear drawback. (Its main advantage is of course that setting up a server to run your app on is trivial, as everything is contained in the release).
What I am wondering: Isn’t the release building a rather heavy procedure (as in, needing a lot of memory and CPU power)? Would this not make the production environment slower during that time?
In my opinion, nobody should be building releases on their production host - they shouldn’t even have build tools installed on that host. You build your releases on your build host, and push those to production and then pull the trigger on rolling it out when you’re ready. This can be entirely automated, automated with some manual intervention, or done entirely manually.
I’ve set up and used different approaches depending on what I’m working on - with Docker/Kubernetes, there were dedicated hosts in the cluster which handled builds, and transfers of those images to the production hosts were super fast because they were colocated. Similarly, with a more traditional setup, we used Jenkins to build releases, and scp them to staging/production hosts and unpack them. My current project, we’re actually shipping virtual machines, so it’s an entirely different situation, but we’re still using releases (which are ultimately installed on the host via rpms).
The release tarballs are certainly a bit fat if you are including ERTS, but if your build and prod hosts are the same, then you don’t need to do that, and can omit ERTS, which is by far the bulk of the baggage the tarball carries.
I agree with you completely, and I think that releases are the path to follow when you have multiple servers and/or you can dedicate time to create your build workflow.
First of all, you need a build environment that suits your needs, which ends up with an extra machine / docker container / something else.
Also, you need to take care of not using Mix.env anywhere inside your production code and including all libraries you are using on the release (i have seen cases of a library that has no application and isn’t included by default on the releases) or your app may crash.
On the other side, you can use docker images with your Erlang version, pre-fetched libraries, pre-compiled application and assets (if you have) for production and just run it using mix, which give you a mostly-accurate snapshot of your software at that point.
Problems of using containers for this? That you will be storing big container images instead of small tarballs. But on the other hand the deployment is easier to maintain and enough for many cases if you don’t run a fail-proof software (no downtimes, only hot code updates) and doing eager loading is not a decisive point.
At the end, I think it depends deeply on each use case: if you have time enough to prepare a build workflow, if your application really needs it, if you like/dislike Dockerized applications…
I have really enjoyed reading the thoughts about this topic, thanks guys!
I suppose one of my main points though is that this is not true. You CAN use a dedicated build environment, but you do not NEED to. If you’re running mix in production, you literally add one command mix release and now you’re using releases, no new build requirements at all.
True, but this isn’t something you should be doing in any project anyway, whether using releases or not.
I do both. I build a docker image containing an erlang release, and then I run that docker image in an ECS cluster. Pushing up the image diff is the same size as pushing up the tarball.
My comparison was between using just releases (without dockerizing) vs using docker images - sorry if I was not clear.
Well, if you are not using Docker to wrap your build and you want to include ERTS, it’s highly recommended to do the build in a non-production machine, as it might be a problem or performance of running application, isn’t it?
Of course! It was just a point because lots of people doesn’t know that
How is your experience about that? One of the main points I see on releases is exactly that: get rid of docker images; just have a tarball with an executable that run on servers with no extra software need, but I haven’t had any docker+releases app deployed.
I can’t speak to docker since I have not used it yet (still on my todo list to try it). However, for running production without containers, you should be using releases. There is a smaller learning curve with Distillery, but its very easy to use once you have invested the couple hours.
I have a number of Elixir and Phoenix apps in production, ranging from Internal and External IT systems (single instance) to several apps that are packaged as rpms and used my many our customers on their locally hosted phone systems. All of these use releases. The newer ones are using Distillery. I have yet up upgrade some from exrm to Distillery, but will be doing that.
I’ve taken a short cut a couple of times. Built and ran with mix on a production and regretted it.
Releases are basically useless for me due to hardcoding config variables. Using docker containers + env vars instead. Ability to tune variables and restart is the must. Ability to launch on different servers with different configuration is an unavoidable everyday routine
Releases have no issue with environment variables. REPLACE_OS_VARS=true, and then in your config just foo: "${FOO}". We use identical images in staging and production, and do all config changes by specifying environment variables in our docker containers.
@thousandsofthem Validation/transformation of the raw config prior to use is best left for init functions in the relevant parts of your application, you have to do validation at these points anyway, I see no reason why one would prefer config.exs for that - I only ever use config.exs to define default values, everything else is handled internally in the application. In my apps, I’ve usually extracted common config logic into a Config module which I then use in init callbacks of my application. You can also use start phases (or just the start callback) to pull config from external systems (for example Vault or something), and do what you need to with it prior to starting your application’s supervisor tree. There are other options as well, such as my conform library, which allows you to do complex validation/transforms of config pulled from either an ini-style config file or from environment variables, and this transformed config is then used by the release.
I often wish that José had gone a different route with config.exs - we wouldn’t be having this conversation, because everyone would just do configuration the way I described above, which is how it’s been done in Erlang for years, and this would be a non-issue, but instead people get used to doing everything in config.exs, and then find that it’s a pain to transition away from it when they want to start using releases - well, yeah, it sucks you have to do that, but it’s not that configuring releases is somehow terrible, because it’s really not, it’s that you have to refactor your code to handle configuration differently. And yes, that’s a valid complaint, but placing the blame on releases is not.
You could accomplish the exact same thing with the following in a release:
defmodule MyApp do
use Application
def start(_type, _args) do
import Supervisor.Spec, warn: false
repo_env = Application.get_env(:indexer_spike, Repo)
configured_env = put_in(repo_env, [:database], ConfigDb.get!("/postgresql/database"))
Application.put_env(:indexer_spike, Repo, configured_env, persistent: true)
children = [
...
]
Supervisor.start_link(children, strategy: :one_for_one, name: __MODULE__.Supervisor)
end
end
In other words, load your configuration from the database and push it into the application env prior to starting your top-level supervision tree. If you have dependencies you need to configure this way, you can add them to :included_applications and place them into your top-level supervisor, or use Application.ensure_all_started(:app) in the start callback.
You don’t have to configure everything in the application start callback either, you can handle individual configuration values in the worker/supervisor init callbacks, where you can do all the validation you’d need to do anyway, and fail the init if the configuration is missing or invalid. Move shared validation logic to a Config module. This is the way it should be done in my opinion. Just because Mix’s config file lets you do this doesn’t mean it’s the right place to do it. Use Mix’s config file to set defaults, use the env-specific configs to set a priori config values (i.e. things you can configure ahead of time because they don’t change from host to host), and do the more complex host and env-specific configuration at runtime, in the application itself - you get all the benefits of writing Elixir code, except now your configuration logic lives next to the thing it’s relevant to. And hey, as a side-effect, you can use releases painlessly!
That’s cool, and TIL that application’s start callback runs before all the dependency applications are started.
But, a couple things:
It’s not nearly as pretty looking as a config file.
That’s some pretty tribal knowledge. Prior to this, I only knew of config/*.exs files to configure applications. Mostly because every library’s documentation show how to do it this way. Even libraries like Ecto’s documentation exclusively show how to configure via config/*.exs.
That would be cool if there was some official support for runtime vs buildtime configuration. Something like config/runtime.exs that if exists, gets merged into the application’s config around the time the application’s start callback is called.
