Cross-Compiling ERTS with OpenSSL for ARM

So, I spent a whole two days trying to get this to work and I’m absolutely stumped at the moment. I need to compile ERTS for an ARM device using a very old arm-linux-gnueabihf compiler. (All of this is an a legacy build container, so the paths are… weird)

My question in short is: Where does the erlang crypto module expect libcrypto.so.1.1 to be, why does it not bundle it when building, despite specifying the OpenSSL path (and why does an old app built with distillery and an old ERTS work fine without libcrypto.so at all?)


I’ve managed to get Open-SSL to compile:

wget https://www.openssl.org/source/openssl-1.1.1d.tar.gz
tar -xzf openssl-1.1.1d.tar.gz 
cd openssl-1.1.1d
./Configure linux-generic32 no-asm shared \
  --prefix="/home/openssl/armbuild" \
  --openssldir="/home/openssl/armfiles" \
  --cross-compile-prefix="arm-linux-gnueabihf-"
make depend && make && make install

when checking the output directory, everything looks alright:

/home/openssl/armbuild/lib/libcrypto.so
/home/openssl/armbuild/lib/pkgconfig/libcrypto.pc
/home/openssl/armbuild/lib/libcrypto.so.1.1 # <--IT'S RIGHT THERE!!
/home/openssl/armbuild/lib/libcrypto.a

Also cross-compiled ncurses-6.2 which worked fine.

Now for compiling erlang/otp, I’m trying the following:

wget http://www.erlang.org/download/otp_src_22.2.tar.gz
tar -zxf otp_src_22.2.tar.gz  
cd otp_src_22.2
export ERL_TOP=`pwd`

./configure --enable-bootstrap-only
make

./otp_build configure \
  --xcomp-conf="$ERL_TOP/xcomp/erl-xcomp-arm-linux.conf" \
  --with-ssl="/home/openssl/armbuild" \
  --enable-dynamic-ssl-lib \
  --without-javac \
  --without-jinterface \
  --without-odbc \
  --without-wx
make -j8
make release

where I modified erl-xcomp-arm-linux.conf to use the correct compiler. This also works and I get an ERTS release. However, the release does not actually bundle libcrypto.

find release/arm-unknown-linux-gnueabihf/ -name '*.so'
release/arm-unknown-linux-gnueabihf/lib/crypto-4.6.3/priv/lib/crypto_callback.so
release/arm-unknown-linux-gnueabihf/lib/crypto-4.6.3/priv/lib/otp_test_engine.so
release/arm-unknown-linux-gnueabihf/lib/crypto-4.6.3/priv/lib/crypto.so
[...]

which leads to Elixir applications built with this ERTS to crash with the following message on boot:

    args: [load_failed,"Failed to load NIF library /TEST/device/lib/crypto-4.6.3/priv/lib/crypto: 'libcrypto.so.1.1: cannot open shared object file: No such file or directory'","OpenSSL might not be installed on this system.\n"]
    format: "Unable to load crypto library. Failed with error:~n\"~p, ~s\"~n~s"
    label: {error_logger,error_msg}

Simply attempting to run erl on the target device crashes with the following message:

{"init terminating in do_boot",{'cannot get bootfile','/TEST/device/bin/start.boot'}}
init terminating in do_boot ({cannot get bootfile,/TEST/device/bin/start.boot})

I’ve tried manually copying or symlinking the .so file to any place the crypto library might expect it but that did not help either. - Ok, this was a last ditch effort.

So yeah, is there any more helpful resources on this topics or any suggestions one could give? I’ve read through the Erlang and OpenSSL [cross-]compilation docs multiple times now and I’m all out of ideas.

Thanks in advance!

To find out where libcrypto is looking for openssl you might use the “ldd” command. On my Linux distribution it shows where a binary expects to find it’s shared libraries.

2 Likes

FWIW the other thing I tried in a similar situation was compiling Erlang with the --disable-dynamic-ssl-lib which I hoped would cause libcrypto to be statically linked into the Erlang crypto, but I was never able to get it to work that I could tell. I am very much hampered by my lack of knowledge in Linux dynamic binding.

1 Like

Oh god… Thanks a ton! I didn’t even realize you could use ldd for checking the shared object dependencies of other shared objects. I finally got it to work!! :slight_smile:

For anyone curious, first of all, I resorted to using openssl 1.0.2o, since that was also present on the device (As @easco described, I also tried using --disable-dynamic-ssl-lib but libcrypto wouldn’t be bundled for me either, I don’t really have an idea what the flag does…)

Next, I used ldd to check out the dependencies of the shared objects generated by the build process for the crypto module. This was the result:

root@device:/TEST$ ldd newbrokenapp/lib/crypto-4.6.3/priv/lib/crypto.so 
        libcrypto.so.1.0 => not found
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6e18000)
        /lib/ld-linux-armhf.so.3 (0xb6f2f000)

root@device:/TEST$ ldd oldworkingapp/lib/crypto-4.3/priv/lib/crypto.so
        libcrypto.so.1.0.0 => /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0 (0xb6dd7000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6ce9000)
        libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xb6cd6000)
        libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xb6cb2000)
        /lib/ld-linux-armhf.so.3 (0xb6f31000)

Ok, well, yeah but why… Turns out the openssl installed on the device that populated the expected default paths was 0.9.8. The updated openssl files from 1.0.2o were put in the cross compiler lib directory… I have no idea why, I didn’t build the fs.

Luckily, the erlang configure supports custom runtime paths, though, so I added this flag --with-ssl-rpath=/usr/lib/arm-linux-gnueabihf and now it’s working.

Thanks again!!

1 Like