How to use preinstalled Erlang on target with releases?

Questions / Help
,
1

So, I have an elixir application that originally came bundled with its own ERTS (in a release) to be run on a specific server.

The original setup was:

  • build server: Docker image in CI, elixir 1.14.4, OTP 25.3 (x86_x64 architecture, Rocky Linux)
  • included ERTS: Built without some libs (like ncurses not present on target server), but same OTP 25.3
  • target server: it is deployed also on a x86_64 architecture but is Ubuntu LTS

This all worked fine until one day the Docker image (which is used by all our build activities, not just the ones related to elixir) started using GLIBC 2.34 and the target server it is deployed to remained at 2.31. Now the ERTS when deployed could no longer find a compatible GLIBC and our release could no longer be started.

So I wanted to solve this by installing Erlang on the target server and no longer bundling ERTS with the release.
include_erts: false,

We verified that Erlang 25.3.2.2 was installed on target server under /lib/erlang, playing around with the erl shell. It seemed to work fine.

But when we unpack the release and do bin/name_of_release start we get this:

{"init terminating in do_boot",{load_failed,[logger_simple_h,supervisor,proc_lib,lists,logger_backend,logger,logger_server,kernel,filename,gen_server,gen,file_server,erl_parse,file_io_server,file,ets,erl_lint,logger_config,erl_eval,application,code_server,code,application_controller,error_handler,heart,logger_filters,gen_event,error_logger,application_master]}}

Basically the whole Erlang libs are missing. It can find erl from PATH just fine, but somehow none of the required OTP.

I tried supplying OTPROOT and ERL_LIBS on the command line while running the script, but to no avail.

It’s built for V13.2 (on Docker) and on target it has V13.2.2.5, same CPU architecture.

Any ideas what’s going on?

Thank you. :slightly_smiling_face:

2

If you’re using docker anyways, why not use a docker image matching your target server to build the release?

1 Like
3

We’re not in charge of the CI and the build servers in that regard, we would have to supply our own way of building with our own Docker image. Not so easy in a big company…

Thank you, though! We might need that as a “last resort,” I will keep it in mind.

4

OK, but does not your CI rely on a Dockerfile that you are supplying? If so, then you can make this work.

5

I faced the same situation and added builds for 2 different ubuntu versions.

6

I have asked the CI team what we can do and what not in this regard, it’s not entirely clear.

The default mode is that we build in a Docker image containing the agreed-on tools. Only this one-size-fits-all doesn’t match my setup.

It remains to be seen if I can actually set up a build with the respective Debian image as base instead. Thanks for the hints. :slightly_smiling_face:

I still don’t understand, though, how even small deviations seem to make it impossible to execute BEAM files on the same HW architecture in the same major version of the ERTS. Something is amiss there…?

7

Your error suggests some misconfiguration rather than some incompatibitly in beam files. kernel is started way before any custom code is loaded, so you never even got to “your code”.

8

That was my original question meant to ask about - what could I do differently to resolve this?

It’s not the local Erlang installation as such. When running erl, kernel and other libraries are found. When running the elixir release there (without including ERTS), it does not find these.

Is there something else I could join in my release to resolve this? Or, the other way around: Could it be something in the release itself that messes up the “search paths” for the Erlang libs?

I found this: Release fails to start when rebar3's include_erts=false is used, 25.0-rc2 · Issue #5826 · erlang/otp · GitHub , so the issue happened to others, too, but I don’t understand what the proposed alternate solution does and how to make it work with my problem. I assume I could just try setting BINDIR when starting the release?

Thanks. :slightly_smiling_face:

9

Are they actually present there? There are few libraries (like crypto for example) that will not be built if their prerequisite (like having a valid openssl installed and linked) are not met. You can easily inspect that by opening the libs folder in the erts installation and looking at the specific libraries folders and their versions.

10

Hello, @D4no0,

$ ls -1 /usr/lib/erlang/lib/
asn1-5.0.21.1
common_test-1.24.0.1
compiler-8.2.6.3
crypto-5.1.4.1
debugger-5.3.1.2
dialyzer-5.0.5
diameter-2.2.7
edoc-1.2
eldap-1.2.11
erl_docgen-1.4
erl_interface-5.3.2.1
erts-13.2.2.5
et-1.6.5
eunit-2.8.2
ftp-1.1.4
inets-8.3.1.2
jinterface-1.13.2
kernel-8.5.4.2
megaco-4.4.3
mnesia-4.21.4.2
observer-2.14
odbc-2.14
os_mon-2.8.2
parsetools-2.4.1
public_key-1.13.3.2
reltool-0.9.1
runtime_tools-1.19
sasl-4.2
snmp-5.13.5
ssh-4.15.3.1
ssl-10.9.1.3
stdlib-4.3.1.3
syntax_tools-3.0.1
tftp-1.0.4
tools-3.5.3
wx-2.2.2.1
xmerl-1.3.31.1

This is the system it would be deployed to. We basically installed all erlang packages LTS 24.04 offers, I guess. Am I sharing the contents of the right folder?

11

OK, but does not your CI rely on a Dockerfile that you are supplying? If so, then you can make this work.

I’m also following up on this and to have an ERTS built based on the same Ubuntu LTS image we are deploying. I assume that if I manage to get this in my build environment for elixir and add it with include_erts that this would resolve the issue - at least how I understood your suggestion.

Tomorrow I’ll know how amenable the tools team will be to my request, but I guess this would be a solution that honors this what I found in the mix docs:

You may also set this option to false if you desire to use the ERTS version installed on the target. Note, however, that the ERTS version on the target must have the exact version as the ERTS version used when the release is assembled. Setting it to false also disables hot code upgrades. Therefore, :include_erts should be set to false with caution and only if you are assembling the release on the same server that runs it.

from: mix release — Mix v1.14.4

12

Looks to be everything in place :thinking: .

The only thing I could recommend more from the top of my head is to actually check your release files, namely:

  • releases/start_erl.data - make sure here your erts version matches
  • releases/0.1.0/elixir - check to make sure once again that erts path version matches for ERTS_BIN

If these are as expected I would still dig in this direction, because most probably some path is messed up, hence why you are missing the libraries. I remember when I first tried to run releases on android I had an issue similar to this, but that was due the fact that elixir releases were using some relative paths that were not resolving correctly.

13

This is the solution my coworker Slawomir Skrzyniarz came up with. It bundles the needed dynamic libraries and patches the executables of the ERTS accordingly to use them. (We’re compiling and deploying on the same architecture, after all.)

We then simply bundle that up in a tarball at the end and unpack it in the right directory on deployment target.

# the base directory here is the path of the release
# as passed to us when registering a function under
# :releases and :steps
mkdir -p lib64
cp -v \
        /usr/lib64/ld-linux-x86-64.so.2 \
        /usr/lib64/libBrokenLocale.so.1 \
        /usr/lib64/libSegFault.so \
        /usr/lib64/libanl.so.1 \
        /usr/lib64/libc.so.6 \
        /usr/lib64/libc_malloc_debug.so.0 \
        /usr/lib64/libdl.so.2 \
        /usr/lib64/libm.so.6 \
        /usr/lib64/libmvec.so.1 \
        /usr/lib64/libnss_compat.so.2 \
        /usr/lib64/libnss_dns.so.2 \
        /usr/lib64/libnss_files.so.2 \
        /usr/lib64/libpthread.so.0 \
        /usr/lib64/libresolv.so.2 \
        /usr/lib64/librt.so.1 \
        /usr/lib64/libthread_db.so.1 \
        /usr/lib64/libtinfo.so.6 \
        /usr/lib64/libutil.so.1 \
        /usr/lib64/libz.so.1 \
        /usr/lib64/libsctp.so.1 \
        /usr/local/gcc-12.2.0/lib64/libgcc_s.so.1 \
        /usr/local/gcc-12.2.0/lib64/libstdc++.so.6 \
        lib64
find bin/ erts-13.2/ -type f -executable -exec patchelf --set-interpreter <absolute installation path on deployment target>/lib64/ld-linux-x86-64.so.2 {} \\; 2>/dev/null
find bin/ erts-13.2/ -type f -executable -exec patchelf --set-rpath <absolute installation path on deployment target>/lib64 {} \\; 2>/dev/null
1 Like