Issue with deploying MJML NIF on Fly.io

I encountered an issue deploying a project using mjml_eex (mjml_eex | Hex) on fly.io. Project works on my machine but not when deploying on Fly.io. The issue seems to be related to not getting the correct precompiled NIF from rustler. The same issue happens with just mjml (mjml | Hex).

I create a new project with the latest Phoenix mix task and the only thing I did was adding {:mjml, “~> 3.0”} as a dependency and deploying it to fly.io with fly launch. Same issue. I tried another rust nif library {:meeseeks, “~> 0.17.0”} and did not have the problem.

Example repo: GitHub - victorbjorklund/example_app

The error message from the machine on fly.io:

2024-03-01T09:46:14Z proxy[9185933a636d08] ams [error]instance refused connection. is your app listening on 0.0.0.0:8080? make sure it is not only listening on 127.0.0.1 (hint: look at your startup logs, servers often print the address they are listening on)
2024-03-01T09:46:14Z app[d891605b400668] ams [info] INFO Main child exited normally with code: 1
2024-03-01T09:46:14Z app[d891605b400668] ams [info] INFO Starting clean up.
2024-03-01T09:46:14Z app[d891605b400668] ams [info] WARN hallpass exited, pid: 306, status: signal: 15 (SIGTERM)
2024-03-01T09:46:14Z app[d891605b400668] ams [info]2024/03/01 09:46:14 listening on [fdaa:0:2e2e:a7b:140:bbee:584c:2]:22 (DNS: [fdaa::3]:53)
2024-03-01T09:46:15Z app[d891605b400668] ams [info][    6.307178] reboot: Restarting system
2024-03-01T09:46:16Z runner[d891605b400668] ams [info]machine did not have a restart policy, defaulting to restart
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]=SUPERVISOR REPORT==== 1-Mar-2024::09:46:16.591703 ===
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    supervisor: {local,kernel_sup}
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    errorContext: start_error
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    reason: {on_load_function_failed,'Elixir.Mjml.Native',
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]                {error,
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]                    {load_failed,
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]                        "Failed to load NIF library: '/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /app/lib/mjml-3.0.2/priv/native/libmjml_nif-v3.0.2-nif-2.16-x86_64-unknown-linux-gnu.so)'"}}}
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    offender: [{pid,undefined},
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]               {id,kernel_safe_sup},
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]               {mfargs,{supervisor,start_link,
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]                                   [{local,kernel_safe_sup},kernel,safe]}},
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]               {restart_type,permanent},
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]               {significant,false},
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]               {shutdown,infinity},
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]               {child_type,supervisor}]
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]=CRASH REPORT==== 1-Mar-2024::09:46:16.591843 ===
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]  crasher:
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    initial call: supervisor:kernel/1
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    pid: <0.1424.0>
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    registered_name: []
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    exception exit: {on_load_function_failed,'Elixir.Mjml.Native',
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]                        {error,
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]                            {load_failed,
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]                                "Failed to load NIF library: '/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /app/lib/mjml-3.0.2/priv/native/libmjml_nif-v3.0.2-nif-2.16-x86_64-unknown-linux-gnu.so)'"}}}
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]      in function  init:run_on_load_handlers/0
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]      in call from kernel:init/1 (kernel.erl, line 196)
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]      in call from supervisor:init/1 (supervisor.erl, line 330)
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]      in call from gen_server:init_it/2 (gen_server.erl, line 851)
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]      in call from gen_server:init_it/6 (gen_server.erl, line 814)
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    ancestors: [kernel_sup,<0.1397.0>]
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    message_queue_len: 0
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    messages: []
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    links: [<0.1399.0>]
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    dictionary: []
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    trap_exit: true
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    status: running
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    heap_size: 987
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    stack_size: 28
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]    reductions: 239
2024-03-01T09:46:16Z app[9185933a636d08] ams [info]  neighbours:
2024-03-01T09:46:17Z app[9185933a636d08] ams [info] WARN Reaped child process with pid: 362 and signal: SIGUSR1, core dumped? false
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]=CRASH REPORT==== 1-Mar-2024::09:46:17.598595 ===
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]  crasher:
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    initial call: application_master:init/4
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    pid: <0.1396.0>
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    registered_name: []
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    exception exit: {{shutdown,
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]                      {failed_to_start_child,kernel_safe_sup,
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]                       {on_load_function_failed,'Elixir.Mjml.Native',
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]                        {error,
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]                         {load_failed,
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]                          "Failed to load NIF library: '/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /app/lib/mjml-3.0.2/priv/native/libmjml_nif-v3.0.2-nif-2.16-x86_64-unknown-linux-gnu.so)'"}}}}},
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]                     {kernel,start,[normal,[]]}}
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]      in function  application_master:init/4 (application_master.erl, line 142)
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    ancestors: [<0.1395.0>]
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    message_queue_len: 1
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    messages: [{'EXIT',<0.1397.0>,normal}]
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    links: [<0.1395.0>,<0.1394.0>]
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    dictionary: []
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    trap_exit: true
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    status: running
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    heap_size: 987
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    stack_size: 28
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    reductions: 158
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]  neighbours:
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]=INFO REPORT==== 1-Mar-2024::09:46:17.604305 ===
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    application: kernel
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    exited: {{shutdown,
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]                 {failed_to_start_child,kernel_safe_sup,
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]                     {on_load_function_failed,'Elixir.Mjml.Native',
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]                         {error,
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]                             {load_failed,
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]                                 "Failed to load NIF library: '/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /app/lib/mjml-3.0.2/priv/native/libmjml_nif-v3.0.2-nif-2.16-x86_64-unknown-linux-gnu.so)'"}}}}},
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]             {kernel,start,[normal,[]]}}
2024-03-01T09:46:17Z app[9185933a636d08] ams [info]    type: permanent
2024-03-01T09:46:18Z app[9185933a636d08] ams [info]{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,'Elixir.Mjml.Native',{error,{load_failed,\"Failed to load NIF library: '/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /app/lib/mjml-3.0.2/priv/native/libmjml_nif-v3.0.2-nif-2.16-x86_64-unknown-linux-gnu.so)'\"}}}}},{kernel,start,[normal,[]]}}}"}
2024-03-01T09:46:18Z app[9185933a636d08] ams [info]Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,'Elixir.Mjml.Native',{error,{load_failed,"Failed to load NIF library: '/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /app/lib/mjml-3.0.2/priv/native/libmjml_nif-v3.0.2-nif-2.16-x86_64-unknown-linux-gnu.so)'"}}}}},{kernel,start,[normal,[]]}}})
2024-03-01T09:46:18Z app[9185933a636d08] ams [info]Crash dump is being written to: erl_crash.dump...done
2024-03-01T09:46:18Z app[9185933a636d08] ams [info] INFO Main child exited normally with code: 1
2024-03-01T09:46:18Z app[9185933a636d08] ams [info] INFO Starting clean up.
2024-03-01T09:46:18Z app[9185933a636d08] ams [info] WARN hallpass exited, pid: 306, status: signal: 15 (SIGTERM)
2024-03-01T09:46:18Z app[9185933a636d08] ams [info]2024/03/01 09:46:18 listening on [fdaa:0:2e2e:a7b:a356:1b51:ec39:2]:22 (DNS: [fdaa::3]:53)
2024-03-01T09:46:19Z app[9185933a636d08] ams [info][    5.270400] reboot: Restarting system
2024-03-01T09:46:19Z runner[9185933a636d08] ams [info]machine did not have a restart policy, defaulting to restart

I’m not sure if this is an issue with Docker, mjml library, fly.io, rustler or what. Any idea on how to get mjml to work on fly.io?

This appears to be an issue with your docker image. Not sure what you’re using, but you need to have an image with glibc 2.33.

Debian buster is 2.28
Debian bookwork is 2.36

I’m not sure if a higher version of glibc will work, but that’s the most likely culprit.

Yea, I googled around and seems no Debian version with 2.33 exists (as far as I could understand. I barely understand what glibc does other than it is used to compile code). After trying a couple of different images I found it works with this Ubuntu image instead of Debian:

“hexpm/elixir:1.16.1-erlang-24.3.4.16-ubuntu-jammy-20240125”

In case someone encounters the same issue in the future.

4 Likes

Hello, ran into the same problem and this helped.
Thanks.

1 Like