Well, now that the app depends on rust, you do the same that you did with git, add rust to the RUN apk
line
(maybe you’ll also need to add cargo, the rust package manager, to the line)
Well, now that the app depends on rust, you do the same that you did with git, add rust to the RUN apk
line
(maybe you’ll also need to add cargo, the rust package manager, to the line)
I tried… I added
RUN apk add --no-cache build-base npm rust
RUN apk add --no-cache build-base npm cargo
to my dockerfile.
but unfortunately it isn’t so easy:
...
Compiling mimalloc v0.1.26
Compiling thiserror v1.0.25
Compiling flatbuffers v2.0.0
error[E0658]: const generics are unstable
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/flatbuffers-2.0.0/src/array.rs:25:35
|
25 | pub struct Array<'a, T: 'a, const N: usize>(&'a [u8], PhantomData<T>);
| ^
|
= note: see issue #74878 <https://github.com/rust-lang/rust/issues/74878> for more information
error[E0658]: const generics are unstable
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/flatbuffers-2.0.0/src/array.rs:27:23
|
27 | impl<'a, T: 'a, const N: usize> Debug for Array<'a, T, N>
| ^
|
= note: see issue #74878 <https://github.com/rust-lang/rust/issues/74878> for more information
error[E0658]: const generics are unstable
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/flatbuffers-2.0.0/src/array.rs:39:23
|
39 | impl<'a, T: 'a, const N: usize> Array<'a, T, N> {
| ^
|
= note: see issue #74878 <https://github.com/rust-lang/rust/issues/74878> for more information
error[E0658]: const generics are unstable
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/flatbuffers-2.0.0/src/array.rs:59:36
|
59 | impl<'a, T: Follow<'a> + 'a, const N: usize> Array<'a, T, N> {
| ^
|
= note: see issue #74878 <https://github.com/rust-lang/rust/issues/74878> for more information
error[E0658]: const generics are unstable
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/flatbuffers-2.0.0/src/array.rs:73:39
|
73 | impl<'a, T: Follow<'a> + Debug, const N: usize> Into<[T::Inner; N]> for Array<'a, T, N> {
| ^
|
= note: see issue #74878 <https://github.com/rust-lang/rust/issues/74878> for more information
error[E0658]: const generics are unstable
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/flatbuffers-2.0.0/src/array.rs:83:36
|
83 | impl<'a, T: Follow<'a> + 'a, const N: usize> Follow<'a> for Array<'a, T, N> {
| ^
|
= note: see issue #74878 <https://github.com/rust-lang/rust/issues/74878> for more information
error[E0658]: const generics are unstable
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/flatbuffers-2.0.0/src/array.rs:91:52
|
91 | pub fn emplace_scalar_array<T: EndianScalar, const N: usize>(
| ^
|
= note: see issue #74878 <https://github.com/rust-lang/rust/issues/74878> for more information
error[E0658]: const generics are unstable
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/flatbuffers-2.0.0/src/array.rs:110:36
|
110 | impl<'a, T: Follow<'a> + 'a, const N: usize> IntoIterator for Array<'a, T, N> {
| ^
|
= note: see issue #74878 <https://github.com/rust-lang/rust/issues/74878> for more information
error[E0658]: const generics are unstable
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/flatbuffers-2.0.0/src/array.rs:120:31
|
120 | pub fn array_init<F, T, const N: usize>(mut initializer: F) -> [T; N]
| ^
|
= note: see issue #74878 <https://github.com/rust-lang/rust/issues/74878> for more information
error: aborting due to 9 previous errors
For more information about this error, try `rustc --explain E0658`.
error: could not compile `flatbuffers`.
To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed
== Compilation error in file lib/explorer/polars_backend/native.ex ==
** (RuntimeError) Rust NIF compile error (rustc exit code 101)
(rustler 0.22.2) lib/rustler/compiler.ex:36: Rustler.Compiler.compile_crate/2
lib/explorer/polars_backend/native.ex:4: (module)
(stdlib 3.15) erl_eval.erl:685: :erl_eval.do_apply/6
could not compile dependency :explorer, "mix compile" failed. You can recompile this dependency with "mix deps.compile explorer", update it with "mix deps.update explorer" or clean it with "mix deps.clean explorer"
The command '/bin/sh -c mix deps.get --only prod && mix deps.compile' returned a non-zero code: 1
The const generics MVP was stabilized in Rust 1.51.
You need to update the image that you’re using, the tutorial on fly.io uses an old version of alpine that has an old rust version than the 1.51 that @cschmatzler mentioned.
Change the first line from FROM hexpm/elixir:1.12.1-erlang-24.0.1-alpine-3.13.3 AS build
to FROM hexpm/elixir:1.12.3-erlang-24.1.4-alpine-3.14.2 AS build
to use a more recent one.
Also, you don’t need to insert new lines to add each new package, apk add
works like apt, dnf, pacman and every other linux package manager, you can do it all in one line.
RUN apk add --no-cache build-base npm git rust cargo
Thank you a lot.
I am a little proud on myself that I have figured that out myself, too.
Now I run into problems that seem not to be connected to Docker. This appears during compiling:
18:54:00.654 [warn] The on_load function for module Elixir.Explorer.PolarsBackend.Native returned:
{:error,
{:load_failed,
'Failed to load NIF library: \'Error relocating /app/_build/prod/lib/explorer/priv/native/libexplorer.so: (null): initial-exec TLS resolves to dynamic definition in /app/_build/prod/lib/explorer/priv/native/libexplorer.so\''}}
and the fly.io logs show:
Recent Logs
2021-11-02T19:25:43.000 [info] exception exit: {{shutdown,
2021-11-02T19:25:43.000 [info] {failed_to_start_child,kernel_safe_sup,
2021-11-02T19:25:43.000 [info] {on_load_function_failed,
2021-11-02T19:25:43.000 [info] 'Elixir.Explorer.PolarsBackend.Native'}}},
2021-11-02T19:25:43.000 [info] {kernel,start,[normal,[]]}}
2021-11-02T19:25:43.000 [info] in function application_master:init/4 (application_master.erl, line 142)
2021-11-02T19:25:43.000 [info] ancestors: [<0.1879.0>]
2021-11-02T19:25:43.000 [info] message_queue_len: 1
2021-11-02T19:25:43.000 [info] messages: [{'EXIT',<0.1881.0>,normal}]
2021-11-02T19:25:43.000 [info] links: [<0.1879.0>,<0.1878.0>]
2021-11-02T19:25:43.000 [info] dictionary: []
2021-11-02T19:25:43.000 [info] trap_exit: true
2021-11-02T19:25:43.000 [info] status: running
2021-11-02T19:25:43.000 [info] heap_size: 376
2021-11-02T19:25:43.000 [info] stack_size: 28
2021-11-02T19:25:43.000 [info] reductions: 165
2021-11-02T19:25:43.000 [info] neighbours:
2021-11-02T19:25:43.000 [info] =INFO REPORT==== 2-Nov-2021::19:25:43.758109 ===
2021-11-02T19:25:43.000 [info] exited: {{shutdrnel
2021-11-02T19:25:43.000 [info] exited: {{shutdown,
2021-11-02T19:25:43.000 [info] {failed_to_start_child,kernel_safe_sup,
2021-11-02T19:25:43.000 [info] {on_load_function_failed,
2021-11-02T19:25:43.000 [info] 'Elixir.Explorer.PolarsBackend.Native'}}},
2021-11-02T19:25:43.000 [info] {kernel,start,[normal,[]]}}
2021-11-02T19:25:43.000 [info] type: permanent
2021-11-02T19:25:44.000 [info] {"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,'Elixir.Explorer.PolarsBackend.Native'}}},{kernel,start,[normal,[]]}}}"}
2021-11-02T19:25:44.000 [info] Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,'Elixir.Explorer.PolarsBackend.Native'}}},{
2021-11-02T19:25:44.000 [info] Crash dump is being written to: erl_crash.dump...done
2021-11-02T19:25:44.000 [info] Main child exited normally with code: 1
2021-11-02T19:25:44.000 [info] Starting clean up.
***v1 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v2
Is anyone able to give me a hint whether that could be an elixir, explorer, rust or docker issue?
My interpretation of the following error message is that the elixir package Explorer cannot “access” Rust in my docker image:
18:54:00.654 [warn] The on_load function for module Elixir.Explorer.PolarsBackend.Native returned:
{:error,
{:load_failed,
'Failed to load NIF library: \'Error relocating /app/_build/prod/lib/explorer/priv/native/libexplorer.so: (null): initial-exec TLS resolves to dynamic definition in /app/_build/prod/lib/explorer/priv/native/libexplorer.so\''}}
I have a hard time understanding the error message initial-exec TLS resolves to dynamic definition
. From my googling TLS could stand for “Transport Layer Security” or “Thread Local Storage”. I guess the later, because it knows a access mode “initial-exec” (IBM Docs). But still no glue why that fails.
btw: Is it possible to test whether Rust works in my docker image?
Any help is very appreciated.
alpine can provide it’s challenges using rustler… try following these docker steps GitHub - avencera/fast_rss: Fast Elixir RSS feed parser, a NIF wrapper around the Rust RSS crate
Thanks, seems similar to what I do, the error remains
My dockerfile:
###
### First Stage - Building the Release
###
FROM hexpm/elixir:1.12.3-erlang-24.1.4-alpine-3.14.2 AS build
# This step installs all the build tools we'll need
RUN apk update && \
apk upgrade --no-cache && \
apk add --no-cache \
git \
curl \
build-base \
libgcc && \
mix local.rebar --force && \
mix local.hex --force
# install rustup
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV RUSTUP_HOME=/root/.rustup \
RUSTFLAGS="-C target-feature=-crt-static" \
CARGO_HOME=/root/.cargo \
PATH="/root/.cargo/bin:$PATH"
# prepare build dir
WORKDIR /app
# extend hex timeout
ENV HEX_HTTP_TIMEOUT=20
# set build ENV as prod
ENV MIX_ENV=prod
ENV SECRET_KEY_BASE=nokey
# Copy over the mix.exs and mix.lock files to load the dependencies. If those
# files don't change, then we don't keep re-fetching and rebuilding the deps.
COPY mix.exs mix.lock ./
COPY config config
# install build dependencies
RUN apk add --no-cache build-base npm
RUN mix deps.get --only prod && \
mix deps.compile
# install npm dependencies
COPY assets/package.json assets/package-lock.json ./assets/
RUN npm --prefix ./assets ci --progress=false --no-audit --loglevel=error
COPY priv priv
COPY assets assets
# NOTE: If using TailwindCSS, it uses a special "purge" step and that requires
# the code in `lib` to see what is being used. Uncomment that here before
# running the npm deploy script if that's the case.
COPY lib lib
# build assets
RUN npm run --prefix ./assets deploy
RUN mix phx.digest
# copy source here if not using TailwindCSS
# COPY lib lib
# compile and build release
COPY rel rel
RUN mix do compile, release
###
### Second Stage - Setup the Runtime Environment
###
# prepare release docker image
FROM alpine:3.13.3 AS app
RUN apk add --no-cache libstdc++ openssl ncurses-libs
RUN apk update && \
apk add --no-cache \
bash \
libgcc \
openssl-dev
WORKDIR /app
RUN chown nobody:nobody /app
USER nobody:nobody
COPY --from=build --chown=nobody:nobody /app/_build/prod/rel/petal ./
ENV HOME=/app
ENV MIX_ENV=prod
ENV SECRET_KEY_BASE=nokey
ENV PORT=4000
CMD ["bin/petal", "start"]
´´´
should match your builder… eg 3.14.2 - but not sure it’ll fix things…
Thanks for pointing that out. But again, doesn’t prevent from the error.
I have now uploaded the entire project to GitHub - Nefcairon/PETALS-Explorer-Boilerplate: Phoenix-Elixir+Tailwind+Alpine+Surface+Explorer.
have a look here: Add cargo to the Docker image · Issue #543 · livebook-dev/livebook · GitHub
maybe musl-dev is the missing piece?
No, sadly not.
I now try to create a Dockerfile without Alpine-Linux. My latest attempts result in following error from fly.io:
2021-11-06T21:03:12.708 app[2c49a8c3] fra [info] 21:03:12.708 [error] Could not find static manifest at "/app/lib/petal-0.1.0/priv/static/cache_manifest.json". Run "mix phx.digest" after building your static files or remove the configuration from "config/prod.exs".
Where should I put RUN mix phx.digest
in my Dockerfile?
This is my current Dockerfile:
# We build rust binaries.
# Rust is required for `explorer` and `polars`, possibly other NIFs.
FROM rust:1.54-slim-buster AS rust
# Stage 1
# Builds the petal release
FROM hexpm/elixir:1.13.0-rc.0-erlang-24.1.4-debian-buster-20210902-slim AS build
ENV RUSTUP_HOME=/usr/local/rustup \
CARGO_HOME=/usr/local/cargo \
PATH=/usr/local/cargo/bin:$PATH
RUN apt-get update && apt-get upgrade -y && \
apt-get install --no-install-recommends -y \
ca-certificates build-essential git && \
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false && \
apt-get clean -y && \
rm -rf /var/lib/apt/lists/*
# Copy Rust compiler, `cargo`, etc:
COPY --from=rust /usr/local/rustup /usr/local/rustup
COPY --from=rust /usr/local/cargo /usr/local/cargo
WORKDIR /app
# Install hex and rebar
RUN mix local.hex --force && \
mix local.rebar --force
# Build for production
ENV MIX_ENV=prod
# Install mix dependencies
COPY mix.exs mix.lock ./
COPY config config
RUN mix do deps.get, deps.compile
# Compile and build the release
COPY rel rel
COPY priv priv
COPY lib lib
RUN mix do compile, release
# Stage 3
# Prepares the runtime environment and copies over the relase.
# We use the same base image, because we need Erlang, Elixir and Mix
# during runtime to spawn the petal standalone runtimes.
# Consequently the release doesn't include ERTS as we have it anyway.
FROM hexpm/elixir:1.13.0-rc.0-erlang-24.1.4-debian-buster-20210902-slim
ENV RUSTUP_HOME=/usr/local/rustup \
CARGO_HOME=/usr/local/cargo \
PATH=/usr/local/cargo/bin:$PATH
RUN apt-get update && apt-get upgrade -y && \
apt-get install --no-install-recommends -y \
# Runtime dependencies
build-essential ca-certificates libncurses5-dev \
# In case someone uses `Mix.install/2` and point to a git repo
git && \
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false && \
apt-get clean -y && \
rm -rf /var/lib/apt/lists/*
# Copy Rust compiler, `cargo`, etc:
COPY --from=rust /usr/local/rustup /usr/local/rustup
COPY --from=rust /usr/local/cargo /usr/local/cargo
# Run in the /data directory by default, makes for
# a good place for the user to mount local volume
WORKDIR /data
ENV HOME=/home/petal
# Make sure someone running the container with `--user`
# has permissions to the home dir (for `Mix.install/2` cache)
RUN mkdir $HOME && chmod 777 $HOME && \
chmod -R a+w $RUSTUP_HOME $CARGO_HOME
# Install hex and rebar for `Mix.install/2` and Mix runtime
RUN mix local.hex --force && \
mix local.rebar --force
# Copy the release build from the previous stage
COPY --from=build /app/_build/prod/rel/petal /app
# Make release executables available to any user,
# in case someone runs the container with `--user`
RUN find /app -executable -type f -exec chmod +x {} +
CMD [ "/app/bin/petal", "start" ]
```
Thanks to the elixir community it’s finally working
If ever somebody runs into similiar problems, here is my Dockerfile. It’s certainly not ideal, but I was able to deploy successfully on fly.io:
# This Dockerfile is based on the Dockerfile from hello_elixir: https://github.com/fly-apps/hello_elixir/blob/main/Dockerfile
#
# Find eligible builder and runner images on Docker Hub. We use Ubuntu/Debian instead of
# Alpine to avoid DNS resolution issues in production.
#
# https://hub.docker.com/r/hexpm/elixir/tags?page=1&name=ubuntu
# https://hub.docker.com/_/ubuntu?tab=tags
#
# This file is based on these images:
#
# - https://hub.docker.com/r/hexpm/elixir/tags - for the build image
# - https://hub.docker.com/_/debian?tab=tags&page=1&name=bullseye-20210902-slim - for the release image
# - https://pkgs.org/ - resource for finding needed packages
# - Ex: hexpm/elixir:1.12.3-erlang-24.1.4-debian-bullseye-20210902-slim
#
ARG BUILDER_IMAGE="hexpm/elixir:1.12.3-erlang-24.1.4-debian-bullseye-20210902-slim"
ARG RUNNER_IMAGE="debian:bullseye-20210902-slim"
FROM ${BUILDER_IMAGE} as builder
# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential npm curl git \
&& apt-get clean && rm -f /var/lib/apt/lists/*_*
# Get Rust
RUN curl https://sh.rustup.rs -sSf | bash -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
# prepare build dir
WORKDIR /app
# install hex + rebar
RUN mix local.hex --force && \
mix local.rebar --force
# set build ENV
ENV MIX_ENV="prod"
# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config
# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile
COPY priv priv
COPY lib lib
# note: if your project uses a tool like https://purgecss.com/,
# which customizes asset compilation based on what it finds in
# your Elixir templates, you will need to move the asset compilation
# step down so that `lib` is available.
COPY assets assets
# For Phoenix 1.6 and later, compile assets using esbuild
# war: -g
RUN npm install tailwindcss postcss autoprefixer postcss-import postcss-nested
RUN cd assets && NODE_ENV=production npx tailwindcss --postcss --minify -i css/app.css -o ../priv/static/assets/app.css && cd ..
RUN npm run --prefix ./assets deploy
# Compile the release
COPY lib lib
RUN mix compile
# Changes to config/runtime.exs don't require recompiling the code
COPY config/runtime.exs config/
COPY rel rel
RUN mix release
# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities
FROM ${RUNNER_IMAGE}
RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales \
&& apt-get clean && rm -f /var/lib/apt/lists/*_*
# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
WORKDIR "/app"
RUN chown nobody /app
# Only copy the final release from the build stage
COPY --from=builder --chown=nobody:root /app/_build/prod/rel ./
USER nobody
# Create a symlink to the application directory by extracting the directory name. This is required
# since the release directory will be named after the application, and we don't know that name.
RUN set -eux; \
ln -nfs /app/$(basename *)/bin/$(basename *) /app/entry
CMD /app/entry start
Hi @Nefcairon,
I was working through this problem yesterday as well.
In the end, I installed nodejs
and npm
packages on the BUILDER
image, changed the order of the COPY
commands and added npm install
.
The NODE_ENV=production npx tailwindcss --postcss --minify -i css/app.css -o ../priv/static/assets/app.css && cd ..
call is in your package.json
file so it will be called when you call mix assets.deploy
.
My final Dockerfile looks like this:
ARG BUILDER_IMAGE="hexpm/elixir:1.12.3-erlang-24.1.4-debian-bullseye-20210902-slim"
ARG RUNNER_IMAGE="debian:bullseye-20210902-slim"
FROM ${BUILDER_IMAGE} as builder
# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential git nodejs npm \
&& apt-get clean && rm -f /var/lib/apt/lists/*_*
# prepare build dir
WORKDIR /app
# install hex + rebar
RUN mix local.hex --force && \
mix local.rebar --force
# set build ENV
ENV MIX_ENV="prod"
# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config
# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile
# Compile the release
COPY lib lib
COPY priv priv
COPY assets assets
RUN cd assets && npm install
RUN mix assets.deploy
RUN mix compile
# Changes to config/runtime.exs don't require recompiling the code
COPY config/runtime.exs config/
COPY rel rel
RUN mix release
# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities
FROM ${RUNNER_IMAGE}
RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales \
&& apt-get clean && rm -f /var/lib/apt/lists/*_*
# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
WORKDIR "/app"
RUN chown nobody /app
# Only copy the final release from the build stage
COPY --from=builder --chown=nobody:root /app/_build/prod/rel ./
USER nobody
# Create a symlink to the application directory by extracting the directory name. This is required
# since the release directory will be named after the application, and we don't know that name.
RUN set -eux; \
ln -nfs /app/$(basename *)/bin/$(basename *) /app/entry
CMD /app/entry start
Hello Andrew,
Will try your version tomorrow, because mine caused
2021-11-22T22:08:35.092 app[8807bc77] fra [info] 22:08:35.092 [error] Could not find static manifest at "/app/petal/lib/petal-0.1.0/priv/static/cache_manifest.json". Run "mix phx.digest" after building your static files or remove the configuration from "config/prod.exs".
in the fly logs.
Will then report here if your version fixed that…
Great news
To make sure you have smaller image, do this for the Rustup step:
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain stable --profile minimal --target x86_64-unknown-linux-gnu -y
The --profile minimal
part is crucial and it shaved off ~500MB from our Docker image. With the default profile Rust is installed as if it’s on a development machine. The minimal profile is for servers / apps.
The --target x86_64-unknown-linux-gnu
thing you can skip but I prefer to be explicit – 50/50, your call.
More info here: Profiles - The rustup book
I am getting error on the npm install, any idea?
------
> [builder 13/18] RUN cd assets && npm install:
#22 0.629 npm ERR! must provide string spec
#22 0.637
#22 0.637 npm ERR! A complete log of this run can be found in:
#22 0.637 npm ERR! /root/.npm/_logs/2022-09-05T15_26_34_548Z-debug.log
------
Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c cd assets && npm install]: exit code: 1