Running Elixir with limited heap size

Hi,

Is it possible to somehow limit the memory usage of the Erlang virtual machine? I’m on a shared server and I don’t have root access.

For example, when I want to run Java I cannot use the default settings:

$ java
Error occurred during initialization of VM
Could not allocate metaspace: 1073741824 bytes

I have to start the JVM with limited heap size:

$ java -XX:MaxHeapSize=512m -XX:CompressedClassSpaceSize=64m -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

$ java -XX:MaxHeapSize=512m -XX:CompressedClassSpaceSize=64m HelloWorld
Hello, World

I am using the new release feature that came in Elixir 1.9. When I try to run my Elixir application I get this error:

$ _build/prod/rel/webapp/bin/webapp start
2019-07-02 15:53:49.850859
    args: [load_failed,"Failed to load NIF library: '/var/www/_build/prod/rel/webapp/lib/asn1- 5.0.9/priv/lib/asn1rt_nif.so: failed to map segment from shared object: Cannot allocate memory'"]
format: "Unable to load asn1 nif library. Failed with error:~n\"~p, ~s\"~n"
label: {error_logger,error_msg}
2019-07-02 15:53:49.852625 crash_report        #{label=>{proc_lib,crash},report=>[[{initial_call,{supervisor,kernel,['Argument__1']}},{pid,<0.976.0>},{registered_name,[]},{error_info,{exit,{on_load_function_failed,asn1rt_nif},[{init,run_on_load_handlers,0,[]},{kernel,init,1,[{file,"kernel.erl"},{line,187}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,295}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[kernel_sup,<0.950.0>]},{message_queue_len,0},{messages,[]},{links,[<0.952.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,265}],[]]}
2019-07-02 15:53:49.852655 supervisor_report   #{label=>{supervisor,start_error},report=>[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{on_load_function_failed,asn1rt_nif}},{offender,[{pid,undefined},{id,kernel_safe_sup},{mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
2019-07-02 15:53:50.858601 crash_report        #{label=>{proc_lib,crash},report=>[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.949.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[<0.948.0>]},{message_queue_len,1},{messages,[{'EXIT',<0.950.0>,normal}]},{links,[<0.948.0>,<0.947.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,193}],[]]}
2019-07-02 15:53:50.861379 std_info            #{label=>{application_controller,exit},report=>[{application,kernel},{exited,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}})

Crash dump is being written to: erl_crash.dump...done

Is it possible to limit the heap size in Elixir/Erlang?

http://erlang.org/documentation/doc-6.1/erts-6.1/doc/html/erl.html

There should be plenty of options available to deal with that, but it seems as if your problem is in a NIF. It won’t be restricted by the VMs Heap.

4 Likes

To limit VM heap size I’ve found this: http://erlang.org/pipermail/erlang-questions/2016-April/088968.html

It says to pass the options +MMsco true +Musac false +MMscs 60 (You find these options here http://erlang.org/doc/man/erts_alloc.html) where 60 is the limit in MB.

I’ve tried it and when the VM reached the limit it crashes…

 eheap_alloc: Cannot allocate 600904 bytes of memory (of type "heap")

@ollran what does it happen to the JVM when it reaches the memory limit?

2 Likes

Thank you @NobbZ and @alvises for your answers. I found that there is _build/prod/rel/webapp/releases/0.1.0/vm.args file where Erlang VM args can be inserted.

I inserted +MMsco true, +Musac false and +MMscs 60 there and got this error:

sl_alloc: Cannot allocate 96 bytes of memory (of type "prepared_code").

I guess that this is so difficult to configure that perhaps it would be better to go with JVM + Clojure this time…

I suppose that it is not easily configurable. I don’t really understand the internals of Erlang VM.

I wrote a test program. It just throws an exception and crashes. :wink:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

I also found this option when reading erl docs:

+hmaxk true|false
Sets whether to kill processes reaching the maximum heap size or not. Default to true.
For more information, see [process_flag(max_heap_size, MaxHeapSize)
(http://erlang.org/doc/man/erlang.html#process_flag_max_heap_size).

http://erlang.org/doc/man/erl.html

Its not at all… Consider it an external program that runs side by side to your erlang program and both are talking via a defined interface. The BEAM can’t control anything about that program except for its lifecycle… Start and stop…

But how restricted is your memory? I had erlang applications starting with less than 100 MB available, I had myself a DO 5$ droplet where the application never used more than 150 MB peak…

Also, as the problem seems to be asn1 related, do you really need it or can you strip that dependency?

4 Likes

I don’t get it. You’re trying to run an app that requires X memory and you have <X available. It doesn’t matter if you’re using JVM or BEAM. You even demonstrated that yourself by testing the same thing with JVM and it crashing.

3 Likes

I think it is about 500MB because I am able to run JVM with these settings:

$ java -XX:MaxHeapSize=500m -XX:CompressedClassSpaceSize=168m HelloWorld

I am just experimenting with a blank Phoenix application. It must be some of Phoenix’s dependencies that uses asn1.

I am just experimenting with a blank Phoenix app because I’d like to get rid of my Node.js application.

No, it was not the same application. It was a test application that was used to test what happens if JVM runs out of memory. I have no problem with JVM, it works fine. I am experimenting with Elixir because I find it very interesting.

Yeah, then I’d say something is up. I’m running a simple Phoenix JSON API in production with a limit of 128MB and it’s using 80MB, never getting close to the limit.

Are there even any nifs shipped with a base phoenix application?

1 Like

Asn.1 is an application bundled with Erlang. In a fresh created project using mix phx.new I can’t find it in the app tree (mix app.tree).

So it has to be included manually.

I tried this:

$ mix phx.new --version
Phoenix v1.4.8
$ mix phx.new test
$ cd test
$ mix app.tree | grep -A 1 -B 8 asn1
│   ├── cowboy
│   │   ├── crypto
│   │   ├── cowlib
│   │   │   └── crypto
│   │   └── ranch
│   │       └── ssl
│   │           ├── crypto
│   │           └── public_key
│   │               ├── asn1
│   │               └── crypto

There seems to be asn1 under the public_key.

1 Like

Okay, you are correct, I checked via an SSH connection from my mobile. It corrected away asn in the grep and I haven’t realised…

Still, a running phoenix application with its dependencies should not blow a 500 MB restriction… Especially not a fresh one without any dependencies…

2 Likes

At this point I’d say you should publish that new app in GitHub so other people can check.

Also include exact Erlang and Elixir versions (putting this in a .tool-versions file in the repo is what I do).

I’ve ran a lot of Phoenix apps and very rarely did any of them surpass 50-60MB of RAM.

1 Like

YES, IT FINALLY WORKS!

I am on MacOS and I cannot install Elixir on the server. So, I am using Docker for cross compiling the project for Debian 8.

So basically I run these commands:

$ docker build .
[...]
Successfully built 65d60d773205
$ docker create 65d60d773205
$ docker cp dca434f3938c415ebb419d3214b5cc3092e2c7ecdeacf03e52ca4e44eac6bde2:webapp.txz .
$ scp webapp.txz me@myserver.com:
$ ssh me@myserver.com
$ tar xvf webapp.txz
$ _build/prod/rel/webapp/bin/webapp start

I only needed +MMscs 60 (thank you @alvises). I commented out the rest.

This is my Dockerfile:

FROM debian:8

WORKDIR /app

RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
RUN apt-get update && \
    apt-get -y upgrade && \
    apt-get -y dist-upgrade && \
    apt-get install -y --no-install-recommends apt-utils && \
    apt-get -y install wget libwxbase3.0-0 libwxgtk3.0-0 libsctp1 xz-utils

RUN wget -O - https://deb.nodesource.com/setup_12.x | bash - && \
    apt-get install -y nodejs

RUN wget https://packages.erlang-solutions.com/erlang/debian/pool/esl-erlang_22.0.4-1~debian~jessie_amd64.deb && \
    wget https://packages.erlang-solutions.com/erlang/debian/pool/elixir_1.9.0-1~debian~jessie_all.deb

RUN dpkg -i esl-erlang_22.0.4-1~debian~jessie_amd64.deb && \
    dpkg -i elixir_1.9.0-1~debian~jessie_all.deb

RUN mix local.hex --force && \
    mix local.rebar --force && \
    yes | mix archive.install hex phx_new 1.4.8

RUN mix phx.new webapp --no-ecto

RUN cd webapp && \
    mix deps.get && \
    mix compile

RUN echo "config :webapp, WebappWeb.Endpoint, server: true" >> webapp/config/prod.exs

RUN cd webapp/assets && \
    npm install && \
    npm audit fix

RUN cd webapp/assets && \
    npx webpack --mode production && \
    cd .. && \
    mix phx.digest

RUN cd webapp && \
    SECRET_KEY_BASE=$(mix phx.gen.secret) MIX_ENV=prod mix release

# The magic happens in here
RUN cd webapp/_build/prod/rel/webapp/releases/0.1.0/ && \
    #echo "+MMsco true" >> vm.args && \
    #echo "+Musac false" >> vm.args && \
    echo "+MMscs 60" >> vm.args

RUN cd webapp && \
    tar cJvf webapp.txz _build && \
    mv webapp.txz ../..

CMD ["webapp/_build/prod/rel/webapp/bin/webapp", "start"]

Thank you, everyone!

1 Like

Your resulting image is unnecessarily bloated.

You probably do not need a full debian:8 under your feets.

Using the builder images by @beardedeagle combined with a multistage build you can get image sizes far less than 100 MB.

Also remember, if done correctly, neither erlang, nor elixir are a runtime requirement.

So in theory you could even use a base image that is compatible to your server, build a release therein, extract that release and only deploy the extracted release to your server. This will probably result in the smallest deployment size.

3 Likes

You may want to try +Meamin. This will disable all the erts memory allocators and fall back to malloc which can sometimes use less memory.

6 Likes