Always use Releases

I haven’t done it in a while but what OP says is correct. It’s basically just mix release and you’re done.

The vibe you’re recalling from LYSE if I recall correctly has more to do with appups/hot upgrades - they should be generally avoided unless you really need the functionality and are prepared to deal with the complexity. Releases are very easy to build and use, and I can’t imagine not using them for deployment. Erlang suffered from poor tooling on that front for a long time, but that’s not been the case for probably 5 years or more.

1 Like

Replying to both of these sentiments, I feel like a lot of the weirdness has been ironed out. Distillery is really easy to use at this point. Even the whole environment variables thing is nice and simple with the support of config :foo, "%{SOME_ENV_VAR}".

I have used releases for one production environment now; a webshop that does not need to be hot-reloaded, but I set up everything so that it is, to ensure that I get a feeling for how it goes.

I agree that using a release seems to be better than the mix run --no-halt. Until now I’ve created every release on my local machine, which meant that moving it to the external server was always somewhat of an issue; a release is a quite large binary file, and as every release is put in its own folder, tools like rsync cannot leverage speedups as you need to move the whole file instead of a diff. So building locally also has a very clear drawback. (Its main advantage is of course that setting up a server to run your app on is trivial, as everything is contained in the release).

What I am wondering: Isn’t the release building a rather heavy procedure (as in, needing a lot of memory and CPU power)? Would this not make the production environment slower during that time?

1 Like

In my opinion, nobody should be building releases on their production host - they shouldn’t even have build tools installed on that host. You build your releases on your build host, and push those to production and then pull the trigger on rolling it out when you’re ready. This can be entirely automated, automated with some manual intervention, or done entirely manually.

I’ve set up and used different approaches depending on what I’m working on - with Docker/Kubernetes, there were dedicated hosts in the cluster which handled builds, and transfers of those images to the production hosts were super fast because they were colocated. Similarly, with a more traditional setup, we used Jenkins to build releases, and scp them to staging/production hosts and unpack them. My current project, we’re actually shipping virtual machines, so it’s an entirely different situation, but we’re still using releases (which are ultimately installed on the host via rpms).

The release tarballs are certainly a bit fat if you are including ERTS, but if your build and prod hosts are the same, then you don’t need to do that, and can omit ERTS, which is by far the bulk of the baggage the tarball carries.

16 Likes

I agree with you completely, and I think that releases are the path to follow when you have multiple servers and/or you can dedicate time to create your build workflow.

First of all, you need a build environment that suits your needs, which ends up with an extra machine / docker container / something else.

Also, you need to take care of not using Mix.env anywhere inside your production code and including all libraries you are using on the release (i have seen cases of a library that has no application and isn’t included by default on the releases) or your app may crash.

On the other side, you can use docker images with your Erlang version, pre-fetched libraries, pre-compiled application and assets (if you have) for production and just run it using mix, which give you a mostly-accurate snapshot of your software at that point.

Problems of using containers for this? That you will be storing big container images instead of small tarballs. But on the other hand the deployment is easier to maintain and enough for many cases if you don’t run a fail-proof software (no downtimes, only hot code updates) and doing eager loading is not a decisive point.

At the end, I think it depends deeply on each use case: if you have time enough to prepare a build workflow, if your application really needs it, if you like/dislike Dockerized applications…

I have really enjoyed reading the thoughts about this topic, thanks guys!

1 Like

I suppose one of my main points though is that this is not true. You CAN use a dedicated build environment, but you do not NEED to. If you’re running mix in production, you literally add one command mix release and now you’re using releases, no new build requirements at all.

True, but this isn’t something you should be doing in any project anyway, whether using releases or not.

I do both. I build a docker image containing an erlang release, and then I run that docker image in an ECS cluster. Pushing up the image diff is the same size as pushing up the tarball.

1 Like

My comparison was between using just releases (without dockerizing) vs using docker images - sorry if I was not clear.

Well, if you are not using Docker to wrap your build and you want to include ERTS, it’s highly recommended to do the build in a non-production machine, as it might be a problem or performance of running application, isn’t it?

Of course! It was just a point because lots of people doesn’t know that :slight_smile:

How is your experience about that? One of the main points I see on releases is exactly that: get rid of docker images; just have a tarball with an executable that run on servers with no extra software need, but I haven’t had any docker+releases app deployed.

I can’t speak to docker since I have not used it yet (still on my todo list to try it). However, for running production without containers, you should be using releases. There is a smaller learning curve with Distillery, but its very easy to use once you have invested the couple hours.

I have a number of Elixir and Phoenix apps in production, ranging from Internal and External IT systems (single instance) to several apps that are packaged as rpms and used my many our customers on their locally hosted phone systems. All of these use releases. The newer ones are using Distillery. I have yet up upgrade some from exrm to Distillery, but will be doing that.

I’ve taken a short cut a couple of times. Built and ran with mix on a production and regretted it.

Releases are basically useless for me due to hardcoding config variables. Using docker containers + env vars instead. Ability to tune variables and restart is the must. Ability to launch on different servers with different configuration is an unavoidable everyday routine

Releases have no issue with environment variables. REPLACE_OS_VARS=true, and then in your config just foo: "${FOO}". We use identical images in staging and production, and do all config changes by specifying environment variables in our docker containers.

7 Likes

Does it support default values? Will custom code (e.g. base64 decode var before assigning) work?

@thousandsofthem Validation/transformation of the raw config prior to use is best left for init functions in the relevant parts of your application, you have to do validation at these points anyway, I see no reason why one would prefer config.exs for that - I only ever use config.exs to define default values, everything else is handled internally in the application. In my apps, I’ve usually extracted common config logic into a Config module which I then use in init callbacks of my application. You can also use start phases (or just the start callback) to pull config from external systems (for example Vault or something), and do what you need to with it prior to starting your application’s supervisor tree. There are other options as well, such as my conform library, which allows you to do complex validation/transforms of config pulled from either an ini-style config file or from environment variables, and this transformed config is then used by the release.

I often wish that José had gone a different route with config.exs - we wouldn’t be having this conversation, because everyone would just do configuration the way I described above, which is how it’s been done in Erlang for years, and this would be a non-issue, but instead people get used to doing everything in config.exs, and then find that it’s a pain to transition away from it when they want to start using releases - well, yeah, it sucks you have to do that, but it’s not that configuring releases is somehow terrible, because it’s really not, it’s that you have to refactor your code to handle configuration differently. And yes, that’s a valid complaint, but placing the blame on releases is not.

7 Likes

We have a configuration database instead of environment vars. This is what I ended up doing:

use Mix.Config

Mix.Task.run "loadpaths"
{:ok, _} = Application.ensure_all_started(:config_db)

config :indexer_spike, Repo,
  adapter: Ecto.Adapters.Postgres,
  database: ConfigDb.get!("/postgresql/database")

So then our builder server can build a single Docker image that can be run in any environment via mix run --no-halt.

You could accomplish the exact same thing with the following in a release:

defmodule MyApp do
  use Application

  def start(_type, _args) do
    import Supervisor.Spec, warn: false

    repo_env = Application.get_env(:indexer_spike, Repo)
    configured_env = put_in(repo_env, [:database], ConfigDb.get!("/postgresql/database"))
    Application.put_env(:indexer_spike, Repo, configured_env, persistent: true)
    
    children = [
      ...
     ]
     Supervisor.start_link(children, strategy: :one_for_one, name: __MODULE__.Supervisor)
  end
end

In other words, load your configuration from the database and push it into the application env prior to starting your top-level supervision tree. If you have dependencies you need to configure this way, you can add them to :included_applications and place them into your top-level supervisor, or use Application.ensure_all_started(:app) in the start callback.

You don’t have to configure everything in the application start callback either, you can handle individual configuration values in the worker/supervisor init callbacks, where you can do all the validation you’d need to do anyway, and fail the init if the configuration is missing or invalid. Move shared validation logic to a Config module. This is the way it should be done in my opinion. Just because Mix’s config file lets you do this doesn’t mean it’s the right place to do it. Use Mix’s config file to set defaults, use the env-specific configs to set a priori config values (i.e. things you can configure ahead of time because they don’t change from host to host), and do the more complex host and env-specific configuration at runtime, in the application itself - you get all the benefits of writing Elixir code, except now your configuration logic lives next to the thing it’s relevant to. And hey, as a side-effect, you can use releases painlessly!

2 Likes

That’s cool, and TIL that application’s start callback runs before all the dependency applications are started. :thumbsup:

But, a couple things:

  1. It’s not nearly as pretty looking as a config file.
  2. That’s some pretty tribal knowledge. Prior to this, I only knew of config/*.exs files to configure applications. Mostly because every library’s documentation show how to do it this way. Even libraries like Ecto’s documentation exclusively show how to configure via config/*.exs.

That would be cool if there was some official support for runtime vs buildtime configuration. Something like config/runtime.exs that if exists, gets merged into the application’s config around the time the application’s start callback is called.

To be clear, the application start callback does not start before all the dependencies in your applications and extra_applications lists. If you have dependencies which are not well-behaved, in other words you need to configure them before starting, then you use included_applications, which puts you in control of the lifecycle of those applications, allowing you to configure them and start them whenever you like.

1.) This is so subjective I don’t think it’s an argument worth making, and rather misses the point I was making above about putting complex configuration validation/transformation logic where it belongs
2.) I absolutely agree that the documentation and guidance on configuration is wildly insufficient, but most everything I brought up is covered in Distillery’s documentation, which includes a page on configuration specifically - but that only helps if you are looking at Distillery’s docs, we need official guidance to cover the same topics.

One thing I have been considering adding to Distillery is support for using config.exs in a release, but it’s not currently implemented yet - in any case, what you are describing here is already how Mix’s configuration file works more or less, the issue boils down to the fact that it does have limitations when used with releases.

2 Likes

Are you using releases on Heroku as well?

2 Likes

I thought config/*.exs is buildtime only configuration. In other words, once the release is built, those configurations are “frozen”.

I was suggesting some config files in the style of Mix.Config (you know, with that nice sexy DSL) that are evaluated at runtime and merged into the application’s configuration at time of the application startup (right before all applications are started).

Does that make sense or am I completely not understand something?

1 Like

You just described how mix config works. It loads all the configs when starting up, before starting any application.

The problem with releases is that mix is no longer there, so it can’t load config when applications start (as it does normally), but the config is serialised when a release is built and later, loaded from the serialised format.