Deployment using distillery and new erlang version fails

Questions / Help
, ,
1

Hello,

So I just setup a new debian machine at work for elixir development. We usually use distillery to create the releases that we then put on the server.

I juts tried to make a new release from this machine and I have runtime errors when launching the binary on ther sever. The only difference between my old debian and the new one is the erlanf erts version

Old one:  Erlang/OTP 21 [erts-10.1.3] 
New one: Erlang/OTP 21 [erts-10.2.4]

On the server when launching the new release I get this with verbose:

 ./bin/prestashopapi foreground --verbosity=verbose

/home/tutu/erts-10.2.4/bin/beam.smp: /lib/x86_64-linux-gnu/libtinfo.so.5: no version information available (required by /home/tutu/erts-10.2.4/bin/beam.smp)
/home/tutu/erts-10.2.4/bin/beam.smp: /lib/x86_64-linux-gnu/libtinfo.so.5: no version information available (required by /home/tutu/erts-10.2.4/bin/beam.smp)
/home/tutu/erts-10.2.4/bin/beam.smp: /lib/x86_64-linux-gnu/libtinfo.so.5: no version information available (required by /home/tutu/erts-10.2.4/bin/beam.smp)
/home/tutu/erts-10.2.4/bin/beam.smp: /lib/x86_64-linux-gnu/libtinfo.so.5: no version information available (required by /home/tutu/erts-10.2.4/bin/beam.smp)
/home/tutu/erts-10.2.4/bin/beam.smp: /lib/x86_64-linux-gnu/libtinfo.so.5: no version information available (required by /home/tutu/erts-10.2.4/bin/beam.smp)
/home/tutu/erts-10.2.4/bin/beam.smp: /lib/x86_64-linux-gnu/libtinfo.so.5: no version information available (required by /home/tutu/erts-10.2.4/bin/beam.smp)
2019-02-28 15:15:14.855123
args: [load_failed,"Failed to load NIF library /home/tutu/lib/crypto-4.4/priv/lib/crypto: 'libcrypto.so.1.1: cannot open shared object file: No such file or directory'","OpenSSL might not be installed on this system.\n"]
format: "Unable to load crypto library. Failed with error:~n\"~p, ~s\"~n~s"
label: {error_logger,error_msg}
2019-02-28 15:15:14.864482 crash_report #{label=>{proc_lib,crash},report=>[[{initial_call,{supervisor,kernel,['Argument__1']}},{pid,<0.1514.0>},{registered_name,[]},{error_info,{exit,{on_load_function_failed,crypto},[{init,run_on_load_handlers,0,[]},{kernel,init,1,[{file,"kernel.erl"},{line,212}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,295}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[kernel_sup,<0.1488.0>]},{message_queue_len,0},{messages,[]},{links,[<0.1490.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,273}],[]]}2019-02-28 15:15:14.865151 supervisor_report #{label=>{supervisor,start_error},report=>[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{on_load_function_failed,crypto}},{offender,[{pid,undefined},{id,kernel_safe_sup},{mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
2019-02-28 15:15:15.876872 crash_report #{label=>{proc_lib,crash},report=>[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.1487.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,crypto}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[<0.1486.0>]},{message_queue_len,1},{messages,[{'EXIT',<0.1488.0>,normal}]},{links,[<0.1486.0>,<0.1485.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,193}],[]]}
2019-02-28 15:15:15.889010 std_info #{label=>{application_controller,exit},report=>[{application,kernel},{exited,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,crypto}}},{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,crypto}}},{kernel,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,crypto}}},{kernel,start,[normal,[]]}}})

Crash dump is being written to: erl_crash.dump...done

Does anyone have any Idea of what could be missing ?

I already installed erlang-crypto and openSSL but that didn’t solve the problem.

2

With distillery, you must compile the project on a machine with the same specs as the machine you will deploy it in.

If for example, you compiled your project on Ubuntu 17.04 and your server has Debian or even Ubuntu 17.10 or 18.04, then the project will fail to run (you will be likely otherwise).

Both machines must also have the same architecture type (x86, etc )

2 Likes
3

Oh ok. That explains a lot. Is there a reason why the machines should have the same specs ?

4

Distillery will compile everything and include the erlang runtime in the package. If the machines are different, then the compiled code wont run on the other machine. See it this way - would a windows app work on a Linux system? Both system are different, same with compiled code for specific machines :smiley:

5

Distillery does allow you to cross compile to a different target machine via the include_erts: parameter. However I haven’t personally tried this.

6

From the book “Elixir in Action”:

I want to stress that the release is no longer dependent on your system’s Erlang and Elixir. It’s fully standalone: you can copy the contents of the _build/prod/rel/todo subfolder to another machine where Elixir and Erlang aren’t installed, and it will still work. Of course, because the release contains Erlang runtime binaries, the target machine has to be powered by the same OS and architecture.

This is possible when you include the erlang runtime, but it has the limitation I mentioned before (must be compiled in machines with the same specs).

1 Like
7

Thanks a lot. Unfortunatly that’s why I do, I copy the _build/prod/rel/api/releases/0.0.1/api.tar.gz to the server.
I also tried to copy the whole subfolder to the server but I still get the errors above. As I said the difference is that the server is using debian jessie and I compiled it on a debian stretch.
But it should normaly use the erlang runtime binaries that aren’t working for some reason…

8

That’s likely the source of your problem :smiley:

9

Yes tha’ts what we guessed. Well, I guess I’ll need to re-install another system then… :woman_shrugging:

10

To clarify for other readers, this phrasing can give the wrong impression. The conflict to watch out for is in the libraries present, not that you happened to use one Linux distribution or another. If the required OS libraries (libc, openssl, etc.) were compatible on both the build host and the deployment host, the release would still run between different distributions just fine. Most people just choose not to try to shave that particular yak, and it’s simplest to target the same OS at both ends. But it was never about the choice of OS specifically, so we shouldn’t phrase it that way when trying to teach others about good practices.

This part in particular was dangerous because it could make people believe that you meant it must have the same number of CPU cores, or same RAM total, or same disk size/speed, etc. That’s not at all true.

Yes, that part’s accurate.

3 Likes
11

True. Point is that different Linux distros (and versions from the same distro) use different libraries. We had this issue when we compiled in Ubuntu 18.04 and tried to deploy in Ubuntu 18.10.

I feel this is being taken out of context :stuck_out_tongue: You care correct in your assessment though, distillery artifacts couldn’t care less about how much RAM a machine has.

12

And I’m saying you misrepresented what caused those incompatibilities, which is fine if it only affects your own understanding, but is harmful in the context of trying to teach others.

One could face similar concerns within the same distribution and version thereof if the underlying libraries changed between hosts. Maybe you’re on a rolling release distro, or haven’t upgraded your build host recently but your deploy hosts have been freshly provisioned. Calling it an OS requirement would give no guidance to a new Distillery user how to resolve such a scenario.

(Don’t use rolling-release distros for production hosts, please. That’s just an illustrative example.)

3 Likes
13

I decided to make a Docker container with the same Debian version as my server to build the releases using distillery and it worked. I don’t really know what was causing an issue since the versions of erlang and phoenix where the same as my host but it works. So I am going to use the Docker container from now on.
Thanks a lot for your help I understand distillery a bit more now :slight_smile:

14

Probably some incompatible version of a dynamically linked library then. Maybe libc or something linked to by a (transitive) dependency?.

15

This is fairly interesting. If you are using Docker, then you don’t really need distillery.
Just have a Docker image with the Elixir version you need and you will remove complexity from your projects (by removing redundant tools)

You can use any of the Elixir Offical images for Docker:

https://hub.docker.com/_/elixir

This way you skip a ton a configurations.

16

I wouldn’t call it a ton of configuration, it took me 10 minutes to set up a distillery release for the first time following the docs in https://hexdocs.pm/distillery/guides/working_with_docker.html. It’s just copy pasting and then figuring out your env variables if you need any.

It does mean a smaller docker image, which can be relevant, but personally I’m just a big fan of the scripts that the Distillery release provides you with, like remote_console.

I realise that a lot of people dislike releases because it used to be a mess (at least that’s what I’ve read). Maybe I was just lucky, but it doesn’t seem like much of a mess anymore.

4 Likes
17

There are also things where mix is more forgiving than building releases. E.g. duplicate modules are not allowed for releases.

3 Likes
18

The Distillery documentation specifically speaks to why releases can still be advantageous, even in the context of a Docker image.

Straight from the horse’s mouth:

That latter bit may make you think that OTP releases aren’t necessary when building with Docker, but they still have considerable value even there: release packages are stripped down, and so produce smaller images; releases are also configured to be automatically set up for distribution, which means they can be easily configured into a cluster, but more importantly they are ready to be remotely administered via remote shell if needed. When deploying source and running via Mix, you have to take extra steps to make sure the node is properly configured for this.

…the application or applications the project defines (and their associated dependencies), are compiled, bundled up together in a single package, and upon startup, loaded and then started in dependency order.

To make this even more explicit, all of your application’s modules are loaded before anything is started, which is not the default behavior for launching with Mix, and that latter approach can introduce one-time delays when a loaded module calls a function on a not-yet-loaded one. The preloaded approach provides more consistent baseline performance in a freshly-launched container.

Anyway, you fundamentally can’t run a project that still depends on Mix to launch, without having the full Elixir and Erlang runtime environment present. This means you’ve got big, inefficient images to sling around, even if a fair amount of the base image can be broadly cached and re-used by the Docker daemon. If you’ve got a highly elastic workload, with underlying hosts coming and going, that cache hit rate can be fairly poor. If you’re using a registry that’s not part of your IAAS provider, such as Docker Hub or Quay, you’ll be paying transit costs on the bytes transferred. If you need to rapidly burst to a higher capacity, you’re at the mercy of the time it takes to pull those hefty images.

A slight adjustment in approach means you can have svelte images that don’t include anything more than what’s necessary for your runtime requirements. With multi-stage builds that have been available in stable Docker releases since 17.05, this is exceptionally easy to accomplish with no external tooling or strange workflows.

A fully-capable image built from a Distillery release on an Alpine base is around 40-60MB total for most of us, while the official Elixir base images are ~90MB for 1.8-alpine and a whopping 1.08GB for 1.8.

I don’t personally think expediency is a worthwhile motive to absorb all of the aforementioned downsides.

4 Likes
19

An incredibly interesting view point, I may created a discussion about this in the future to cement more knowledge, since we are actually debating this topic in my company :smiley:

1 Like