What do you need from a deployment tool?

[DianaOlympos]

A build server that enables to build easily an elixir release for any system, cross compiled if needed, with no outside perturbation, that allows you to output an artifact and that would, in theory, be flexible enough to allow plugins for the final artifact (Docker container, deb or RPM bundle, tarball, etc)

Yes this is in theory a solved problem… but the more we looked at what existed in CircleCI and co, the less it seemed solved. Concourse was interesting but ends up rebuilding everything ourselves…

And we can use it as a learning experience about DevOps and building things for our team.

[dch]

I’ve not listened to the talk yet but here’s a couple of things I’d like:

• dump the crufty layers of shell scripts (5+ I think) that the underlying erlang releases run before calling erlexec. See a simpler example below
• something that can itself be distributed as a .ez / escript archive for simplicity
• the tool generates a release, and offers a pluggable system to wire it up to your puppet/chef/ansible/shell scripts for deployment

Almost all of the deployment stuff in any reasonable system is going to be custom - update load balancers, checkpoint dbs, run migrations, shuffle code loading and updates across servers, moving in & out of load balancers as we go. There are existing tools in your infrastructure already to deal with that, no need to reimplement it all.

#!/bin/sh
# a minimal shell script that runs your elixir package directly without intervening
# relx/distillery generated shell scripts. it doesn't have HEART support here, 
# well because it wouldn't be *minimal* anymore, would it? 
ERTS=erts-8.3.5.1

# essential
export PATH=/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
export BINDIR=/usr/local/lib/erlang/$ERTS/bin
export PIPE_DIR=/var/run/halbot/erl_pipes/mrbrown_halbot@127.0.0.1/

export ERL_CRASH_DUMP=/dev/null
export ERL_CRASH_DUMP_BYTES=0

export RUN_ERL_DISABLE_FLOWCNTRL=1
# elixir

export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
export LD_LIBRARY_PATH=/usr/local/lib/erlang/$ERTS/lib:

# your app
NAME=mybot
VERSION=0.1.9
ERL_LIBS=$APP_DIR/lib
export ROOTDIR=/usr/local/lib/$NAME
export HOME=/var/run/$NAME
LOG_DIR=/var/log/$NAME
export RELEASE_LOG_DIR=$LOG_DIR
export RUNNER_LOG_DIR=$LOG_DIR
cd $ROOTDIR

# run_erl $PIPE_DIR $LOG_DIR
$BINDIR/erlexec -init_debug \
	-boot $ROOTDIR/releases/$VERSION/$NAME \
	-boot_var ERTS_LIB_DIR /usr/local/lib/erlang/lib \
	-env ERL_LIBS $ERL_LIBS \
	-pa /usr/local/lib/mybot/lib/$NAME-$VERSION/consolidated \
 	-mode embedded \
	-args_file /usr/local/etc/mybot/vm.args \
	-config /usr/local/etc/mybot/sys.config \
	-start_epmd false \
	-user Elixir.IEx.CLI \
	-extra \
--no-halt +iex -- $1

[brightball]

Could I submit @bitwalker’s application for him?

[andre1sk]

A lot of answers seem to focus on “you” already have infrastructure in place and ops team to handle this stuff. I think people tend to forget that not everyone does. Especially people trying out Elixir for side and smaller projects. Just because I have dedicated ops team on my proj at work that have setup a very sophisticated deployment pipeline does not blind me to the fact that there are a lot of cases were people do not have that.

[bitwalker]

dump the crufty layers of shell scripts (5+ I think) that the underlying erlang releases run before calling erlexec.

I’d be the first one to say I’m not a fan of needing all the shell scripts, but I’ll defend myself here by saying that they are both necessary and “crufty” for good reasons. Releases need to be portable, that means using POSIX shell where possible (although there are some current things requiring the use of bash that I’d like to get rid of, but have not yet found a good solution for). They also need to support custom behaviour pre/post certain stages of the release lifecycle (configuration, stopping/starting, upgrading/downgrading). There is also a need for supporting custom commands (extensions) to the release script for applications which may be consumed as services (think something along the lines of service myapp status). Additionally, the “layers” as you put them are in place because they are needed to support hot upgrades, your minimal script doesn’t support the vast majority (well, any) of these things, which is fine for a specific application which doesn’t need any of them, but isn’t really useful for a general tool. Furthermore, the shell scripts exec into each other, so this isn’t a stack of subshells, but a single process. There are currently 3 layers before erlexec - the initial “check” script which makes sure bash is available; the “boot loader” which determines which release to run (in the case of multiple versions being available, such as is common when using upgrades) as well as determining whether to use the new OTP 20 signal handler or a custom trap to deal with signals properly and setting up important environment variables; and finally the “boot” script itself, which dispatches commands to the appropriate handler (each of which lives in their own script for easier maintenance, so I suppose you could say these represent a fourth layer). I’m a huge advocate of keeping things as simple as possible, both for ease of maintenance as well as flexibility, but the harsh reality is that there is a lot of complexity in supporting the wide variety of deployments with releases, and the scripts that have evolved have done so in order to support that complexity in the simplest way possible. There are still a number of improvements I have in mind in this area, but I think it’s all too easy to assume it’s all “cruft” because you aren’t using it, when there are many organizations who need many if not all of these things. I personally prioritize making the boot process as reliable and fast as possible, so if you feel that the extra layers are causing issues, please open an issue so we can discuss how to solve those problems. If I don’t hear from the community on something, I can’t do much to help - I operate based on feedback and my own experiences deploying Elixir applications, and that may not necessarily match your needs or experiences, so I want to hear about those things.

Something that can itself be distributed as a .ez / escript archive for simplicity

The closest thing to this is the “executable releases” feature I added to Distillery awhile back, but all this it is, is a self-extracting tarball containing the normal release bits, with some extra options to support cleaning up the extracted artifacts post-execution if desired. There is no way to support releases in an escript archive, and I personally believe that any solution not built on releases is throwing away a couple decades worth of engineering expertise and experience from Erlang, just to reinvent the wheel, so I don’t believe this is an avenue for solving deployment in Elixir generally, though it may certainly work for a subset of tools and services, but that tooling is already in place, and comes with certain requirements (such as Erlang being installed on the target system). I think it is far more profitable to upstream changes to OTP that we can leverage in Elixir if we see that there are things holding us back. I do believe there is room for improvement with the release handling bits, but without time to research solutions, we have to work with what we have at the moment. I think in the near term, solutions need to build on that, but remain modular/flexible enough to take advantage of any improvements that come down the pipe.

the tool generates a release, and offers a pluggable system to wire it up to your puppet/chef/ansible/shell scripts for deployment

As far as I can tell from your description, this is absolutely already in place. You can use plugins in Distillery (using the after_package callback) to execute arbitrary Elixir code with a release all built. You could use this to call whatever external tool you want, or simply use Elixir to orchestrate the deployment. If there is something missing here though, I’d love to hear about it.

Almost all of the deployment stuff in any reasonable system is going to be custom - update load balancers, checkpoint dbs, run migrations, shuffle code loading and updates across servers, moving in & out of load balancers as we go. There are existing tools in your infrastructure already to deal with that, no need to reimplement it all.

I totally agree with this - whatever tools the community evolves need to essentially form solid and flexible primitives which can fit neatly into any given build/deployment pipeline. We just need to make sure those primitives are easy to use, compose well, and are well documented. Additional tooling can also be built to solve use cases where you don’t have any of that infrastructure (and is essentially the role tools like edeliver, bootleg, etc. fill).

[cmkarlsson]

@andre1sk

A tool that makes a basic case easy would not be a bad thing to have.

This is what I mostly do for my simple web apps which I deploy to a VPS.

1. Build a release using destillery which gives you a self-contained release which you can zip up.
2. Copy tar-ball to server
3. Extract to a good location
4. Start or upgrade through provided start up script.

I agree that 1. might need better documentation to avoid the most common mistakes. I can’t see that 2, 3 or 4 would be a problem? Simple bash script would do it.

@DavidAntaramian

When I first started using Elixir, I often heard that deployment was an issue, and I started repeating that as if it was a mantra. The company I worked with at the time started by deploying to Heroku, and that worked well for our purposes. I moved companies, and my current company was also deploying Elixir to Heroku. As our needs grew, I learned how to build releases with Distillery. I put together a deployment pipeline using non-Elixir tools and started shipping releases from our build servers through to staging and production.

And now I question what this “deployment is an issue” comment really means.

Exactly this! I agree completely with your post and conclusion.

[cmkarlsson]

@bitwalker

and I personally believe that any solution not built on releases is throwing away a couple decades worth of engineering expertise and experience from Erlang, just to reinvent the wheel, so I don’t believe this is an avenue for solving deployment in Elixir generally, though it may certainly work for a subset of tools and services, but that tooling is already in place, and comes with certain requirements (such as Erlang being installed on the target system). I think it is far more profitable to upstream changes to OTP that we can leverage in Elixir if we see that there are things holding us back.

Completely agree here. With erlang it is very important to ask the question why erlang has solved the problem the way it did. Things may seem odd and not according to modern standard but most of them are there for a very valid reason and they have been used for decades in critical environments.

Yes, I agree things can likely be streamlined to work better but starting from scratch is likely not the solution here.

[andre1sk]

@cmkarlsson

Build a release using destillery which gives you a self-contained release which you can zip up.

Right provided your vps and your dev box are compatible

[cmkarlsson]

Perhaps I am making too many assumptions here?

The old best practice of “Never build software on your developer machine and always build on the same architecture as you are deploy to” is perhaps not as widely known as I think?. Back in the days it meant one physical server per deployment target, then along came Virtual Machines which made life easier and now we have docker which of course makes it even easier. But if you don’t know this I guess the barrier gets higher straight away.

Learning how to use one of the above technologies and integrate them into your development process can be hard and time consuming.

For me this is sort of a natural thing in development like using version control and writing tests but perhaps it is more of an obstacle than I think?

If it is a problem (and it may very well be) the question is how to solve it? Because that lots of external components are needed (docker/virtual machines/physcial servers) I think it is best addressed with documentation. There could be official examples how to setup this up to build releases on your target platform. Creating a tool to do all of this must do lots of choices which may not work with lots of shops that use different build and deploy environments

[andre1sk]

You is very relative here. For someone coming from something like say Node

1. Setup Docker
2. Setup Distillery
3. Configure everything
4. Build a release
   …

Will be a bit involved. None is saying a magic black box needs to be created that will replace Terraform, Ansible, rundeck or whatever other instruments projects are using in their deploy pipeline. But having simple out of the box option would go a long way to making on-boarding new less experienced devs easier.

[brightball]

Some of this comes with the territory of running on your own servers too. Gigalixir is stepping into the space to provide an option. We forget that Ruby was a pain in the tail to host yourself and that Heroku, Engine Yard, etc started almost entirely to grease that squeaky wheel. For years Heroku has had forced daily restarts just to deal with memory leaks.

Configuring Apache or nginx for other C based languages had its own set of concerns. Configuring Java servers and then building out war files with XML config scripts had some steps involved too.

It’s more of a recent trend to see the self contained Node or Go and a whole lot of engineering went into making it possible to compile Go for multiple different architectures from a single machine.

Maybe it makes sense to provide cross platform code to streamline the process of doing a git pull -> build on the target machine (similar to the git push -> build experience with Heroku). For single machine projects starting out this will simplify things and the capabilities are available for more advanced options later when you need them…just an idea.

[bitwalker]

One of the things I’ve been evaluating recently is following the Nerves example a bit, and using buildroot as a cross-compilation toolchain, but stopping short of generating the root filesystem image. This would allow targeting arbitrary platforms from any machine, which in my opinion is the primary pain point at this time.

This still comes with some degree of burden on the user; you need to specify the target platform and the system libraries your application needs, the tooling needs to initialize the toolchain which can take time (but there are some options we have for reducing that as much as possible), and I’m not yet certain how easy it is to specify arbitrary versions of OTP, but if a build is required, that too takes time the first time it is done. I could generate toolchains for the common platforms and packages for each for the current and last versions of OTP to speed up a lot of that, but that comes with it’s own concern (such as hosting/CDN for the packages). The Nerves team has solved some of this, and I have yet to talk with them about sharing some of the infrastructure around it, since that makes a lot of sense in my mind (for instance Nerves caches toolchains, and we could reuse that cache, but the problem there is that it forms an implicit dependency on Nerves implementation, where some kind of shared plumbing for that would ensure that we aren’t stepping on each others toes).

From my research so far, we could get things pretty streamlined for this setup, but it needs to be opt-in, since many organizations already have build infrastructure in place, so all of the above does nothing to improve their situation. Part of me wishes I could take it a step further and actually generate the filesystem image and use that as the build artifact, since it would be very similar to deploying a container, without actually requiring any container infrastructure, but deploying those images requires mounting the filesystem image, binding network interfaces to it, etc., and I’m not sure what all of the “gotchas” are there, but I suspect it just shifts the burden rather than eliminating it.

In my dreams, I long for something as simple as golang deployments, but ERTS just simply isn’t designed to be deployed as a single binary. I haven’t been able to determine if it’s even possible to statically link ERTS with all of the system libraries it needs as well as NIFs from your dependencies so that you could at least deploy a release to a blank host and have everything just work, but even then that would imply that you would need to do a fresh build of ERTS for each application and that just isn’t practical. Go is designed around embedding the runtime in the executable for an application, but OTP is designed to be built once and then interpret the bytecode for arbitrary applications - it’s unreasonable to assume that we can achieve parity with differences like that. I think it’s certainly theoretically possible that an alternate implementation of BEAM could be designed to be embedded in an executable, perhaps storing the bytecode for applications as a resource, but doing so you would need to sacrifice hot upgrades (which is certainly fine for a large number of deployments), not to mention there is no such alternate implementation, so it’s purely hypothetical.

So in the end, we have some “hard” constraints we have to accept, I think at this point the conversation needs to be more along the lines of “should deployment in Elixir be ‘easy to use’, ‘easy to learn’, and/or ‘easy to use without learning’”. I don’t think “easy to use without learning” is a reasonable goal for this, so figuring out as a community what “easy to use” and “easy to learn” means and then trying to get as close to those goals as possible seems like the right path forward. I think it’s also important to clarify that “easy to use” doesn’t mean that you don’t have to learn things, just that the tool is intuitive, consistent, and flexible enough to stay out of your way as much as possible; “easy to learn” is a much harder goal, and I suspect the one which has/will generate the most debate.

[DianaOlympos]

On statically link of everything, this is something i will probably explore in the coming months. In case you want to exchange ideas.

I would especially like providing a build tooling that use Distillery but provides static linking through musl and build a tarball with all needed dependencies (OpenSSL comes to mind), so that we have a “fully ready” app outputted. But i think that should be pluggable in that system, so that people can choose easily. I have ideas and begun playing with it. It will at least ease the deployment, but will probably not totally solve it. But it should also make cross compilation easier…

About mounting a filesystem image : i am not sure yet. There are stuff we could do and i have ideas. An Illumos Zone environment would make things easier, we could produce a Zone image… We could also probably provide a Nix-env that work well. Guix could also work.

For containers, it is a bit harder. But it could be possible to side load something like Oracle does with https://github.com/oracle/crashcart … or do something like micro container, but this is, i think, more an output plugin problem than a solution for everyone.

[bitwalker]

On statically link of everything, this is something i will probably explore in the coming months. In case you want to exchange ideas.

I would be happy to collaborate! Might be a good idea to open an issue on the Distillery tracker where we can talk through ideas and experiment a bit, there are various other members of the community watching the tracker, including those who contribute to not only Distillery but Elixir and Erlang, and I tend to pay closer attention to it myself (for obvious reasons), but a thread dedicated to it here may be more visible, so I could see having the discussion in both places as beneficial.

I would especially like providing a build tooling that use Distillery but provides static linking through musl and build a tarball with all needed dependencies (OpenSSL comes to mind), so that we have a “fully ready” app outputted.

Yeah this is more or less what I had in mind, but I have not tried an actual implementation of this to see how well it works in practice, and what the pain points are. I just found the docs that indicate how to statically link NIFs not in the standard library (as well as any libs they depend on), but I wonder if there are other issues hiding there (is there any special support required to ensure such NIFs support being statically linked?). If NIFs are statically linked, then that means each application which uses NIFs requires a unique build of ERTS, and with how long that takes to build, that may be really painful. I don’t know if it’s possible to blend static and dynamic linking to resolve that, but even if it was, I think you sacrifice all the benefits of a static build if you still need all the system libraries available for dynamically linked NIFs, so my intuition tells me this is a dead end simply because of the build overhead required, but I would be interested to see the results of some experiments to validate that assumption.

But i think that should be pluggable in that system, so that people can choose easily. I have ideas and begun playing with it. It will at least ease the deployment, but will probably not totally solve it. But it should also make cross compilation easier…

I would agree that this should definitely be an “optional” path, as there are clearly much simpler ways to operate assuming you have some basic build infrastructure already in place, so this is the kind of thing you would opt into if you want to avoid all of that and build and deploy straight from your development machine (or perhaps use existing build infra that doesn’t match your target system, and so needs cross-compilation).

About mounting a filesystem image : i am not sure yet. There are stuff we could do and i have ideas. An Illumos Zone environment would make things easier, we could produce a Zone image… We could also probably provide a Nix-env that work well. Guix could also work.

Oh I know for sure it works, and as you have pointed out, one could use zones, LXC, etc., to provide platform-specific outputs; the question from my point of view is “who builds those plugins?”. For example, while I would love to work in an environment with Illumos, I don’t, so I don’t have the operational expertise to build a solution for zones which makes the most of their capabilities. I could research and develop that expertise, but having to do that for all possible environments is not practical. So what I would like to bulid are really solid base primitives which fulfills those requirements I mentioned previously (“easy to use” and “easy to learn”), and let the community develop plugins which build on those primitives in a way that is seamless (i.e. add dependency A providing the plugin for zones, put a line in your configuration to use that plugin, and everything just works). To some extent this has already happened, there are a number of plugins for Distillery to produce things like .rpm or .deb packages, and there are higher level tools like Edeliver and Bootleg which also build on Distillery’s output, but the underlying pain points (being unable to build from any machine and deploy to any target, configuration discrepancies, no Mix, etc.) bubble up to those plugins and tools, so we need to solve for those at the lowest level, and only then will higher level tools be free of having to hack around those issues.

For containers, it is a bit harder. But it could be possible to side load something like Oracle does with GitHub - oracle/crashcart: CrashCart: sideload binaries into a running container … or do something like micro container, but this is, i think, more an output plugin problem than a solution for everyone.

Right, there are already tools for building container images from a release, so I think this is more or less a “solved” problem, or at least not a pain point for the majority of the community (and speaking from experience, working with containers with what we have today is already pretty painless, just using Distillery and raw Docker commands, so I think the majority of issues lie elsewhere).

[orestis]

Maybe I’m clueless, but I don’t see what’s the fuss about single/self-contained binaries. My impression is that with OTP releases we get self-contained directories, which you can zip up and install on a machine with no dependencies whatsoever.

Is it because there is the need of providing a different ERTS on different platforms e.g. Linux/Darwin/Windows? If so, then I don’t think a deploy tool in Elixir is the right level of abstraction to attack this. I would imagine that doing this kind of work in collaboration with the OTP team and being able to provide a cross-platform ERTS would result in less complexity all around.

Of course, I’ve no idea what I’m talking about and I don’t mean to discourage anyone from doing work they like and need, and I’m infinitely grateful to @bitwalker and everyone else that works on making Elixir/Erlang easier to use.

[OvermindDL1]

In general an ERTS needs to work on the same version of erlang as it was built for.

In addition, a lot of libraries compile code for NIF’s or Ports or so, and those are quite often platform dependent.

[Crowdhailer]

I would like to see one deployment tool that made a bunch of assumptions about how deployment was going to be done. So many assumptions that most of the more capable developers here would find it restrictive. It would act as a starting point.

When it did not fulfil some users requirements another solution can be created or this original tool extended where possible.
I am working on something myself with the following assumptions.

• Docker is used
• Configuration is with ENV variables
• Clustering will have to deal with dynamic ip addresses

I recently posted an intro talk to setting up projects, there is a follow up in the works which is more thorough an covers a decent deployment solution.

@talentdeficit

writing a ‘generic’ tool to target bare metal, vps, aws, heroku, k8s, mesos, gce, azure, digital ocean, etc is a herculean task and likely a fool’s errand

Agreed. This problem would be far less if starting from any sensible set of defaults and expanding in a pragmatic fashion.

@jesse

Another common problem is using System.get_env/1 in prod.exs instead of Distillery’s ${VAR_NAME} and setting REPLACE_OS_VARS=true.

Something I tripped up on myself. Always using env is restrictive but works for a lot of cases, I have gone so far as to remove config.exs from my project generator so I have only one way to manage runtime config.

@andre1sk

You is very relative here. For someone coming from something like say Node

1. Setup Docker
2. Setup Distillery
3. Configure everything
4. Build a release
   …

Will be a bit involved

I’ve managed to get starting up a new Elixir project down to 3 commands, without even having Elixir installed (CAVEAT on a machine with docker. My target is people interested in the polyglot nature of microservices). My hope is that with a few more command a multi node deployment on production machines can be created.

I will probably assume deployment to kuberneties. I may even assume one particular platform, perhaps gigalixir. I hope this can be expanded in time, but is not something to start with.

[andre1sk]

If we assume k8s one could probably come up with a way to build a Helm chart at least for Phoenix apps and it will be a 2 command deploy. Now k8s is a fairly strong req.

[hubertlepicki]

@jesse

server: true in prod.exs

Any idea why this is not there by default? I just wasted an hour before I realized it’s not in here.

[Gazler]

Default `server: true` in `prod.exs`

@chrismccord said:

It’s not set for prod by default because you may want to run other tasks, for example mix ecto.migrate or even iex to debug prod and in those cases you won’t want to spin up a phoenix server.