Running Elixir with limited heap size

ollran

Hi,

Is it possible to somehow limit the memory usage of the Erlang virtual machine? I’m on a shared server and I don’t have root access.

For example, when I want to run Java I cannot use the default settings:

$ java
Error occurred during initialization of VM
Could not allocate metaspace: 1073741824 bytes

I have to start the JVM with limited heap size:

$ java -XX:MaxHeapSize=512m -XX:CompressedClassSpaceSize=64m -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

$ java -XX:MaxHeapSize=512m -XX:CompressedClassSpaceSize=64m HelloWorld
Hello, World

I am using the new release feature that came in Elixir 1.9. When I try to run my Elixir application I get this error:

$ _build/prod/rel/webapp/bin/webapp start
2019-07-02 15:53:49.850859
    args: [load_failed,"Failed to load NIF library: '/var/www/_build/prod/rel/webapp/lib/asn1- 5.0.9/priv/lib/asn1rt_nif.so: failed to map segment from shared object: Cannot allocate memory'"]
format: "Unable to load asn1 nif library. Failed with error:~n\"~p, ~s\"~n"
label: {error_logger,error_msg}
2019-07-02 15:53:49.852625 crash_report        #{label=>{proc_lib,crash},report=>[[{initial_call,{supervisor,kernel,['Argument__1']}},{pid,<0.976.0>},{registered_name,[]},{error_info,{exit,{on_load_function_failed,asn1rt_nif},[{init,run_on_load_handlers,0,[]},{kernel,init,1,[{file,"kernel.erl"},{line,187}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,295}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[kernel_sup,<0.950.0>]},{message_queue_len,0},{messages,[]},{links,[<0.952.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,265}],[]]}
2019-07-02 15:53:49.852655 supervisor_report   #{label=>{supervisor,start_error},report=>[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{on_load_function_failed,asn1rt_nif}},{offender,[{pid,undefined},{id,kernel_safe_sup},{mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
2019-07-02 15:53:50.858601 crash_report        #{label=>{proc_lib,crash},report=>[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.949.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[<0.948.0>]},{message_queue_len,1},{messages,[{'EXIT',<0.950.0>,normal}]},{links,[<0.948.0>,<0.947.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,193}],[]]}
2019-07-02 15:53:50.861379 std_info            #{label=>{application_controller,exit},report=>[{application,kernel},{exited,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}})

Crash dump is being written to: erl_crash.dump...done

Is it possible to limit the heap size in Elixir/Erlang?

14 comments

#beam #memory #mix-release

0 5699 14

2019-07-04 08:31:56 UTC

Showing Posts 1 to 10

Show Best Posts
Show All (oldest first)
Show All (newest first)

NobbZ

There should be plenty of options available to deal with that, but it seems as if your problem is in a NIF. It won’t be restricted by the VMs Heap.

Post #1

alvises

To limit VM heap size I’ve found this: [erlang-questions] Max heap size

It says to pass the options +MMsco true +Musac false +MMscs 60 (You find these options here erts_alloc — OTP 29.0.2 (erts 17.0.2)) where 60 is the limit in MB.

I’ve tried it and when the VM reached the limit it crashes…

 eheap_alloc: Cannot allocate 600904 bytes of memory (of type "heap")

@ollran what does it happen to the JVM when it reaches the memory limit?

Post #2

ollran

Thank you @NobbZ and @alvises for your answers. I found that there is _build/prod/rel/webapp/releases/0.1.0/vm.args file where Erlang VM args can be inserted.

I inserted +MMsco true, +Musac false and +MMscs 60 there and got this error:

sl_alloc: Cannot allocate 96 bytes of memory (of type "prepared_code").

I guess that this is so difficult to configure that perhaps it would be better to go with JVM + Clojure this time…

I suppose that it is not easily configurable. I don’t really understand the internals of Erlang VM.

I wrote a test program. It just throws an exception and crashes.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

I also found this option when reading erl docs:

+hmaxk true|false
Sets whether to kill processes reaching the maximum heap size or not. Default to true.
For more information, see [process_flag(max_heap_size, MaxHeapSize)
(http://erlang.org/doc/man/erlang.html#process_flag_max_heap_size).

Post #3

NobbZ

Its not at all… Consider it an external program that runs side by side to your erlang program and both are talking via a defined interface. The BEAM can’t control anything about that program except for its lifecycle… Start and stop…

But how restricted is your memory? I had erlang applications starting with less than 100 MB available, I had myself a DO 5$ droplet where the application never used more than 150 MB peak…

Also, as the problem seems to be asn1 related, do you really need it or can you strip that dependency?

Post #4

jola

I don’t get it. You’re trying to run an app that requires X memory and you have <X available. It doesn’t matter if you’re using JVM or BEAM. You even demonstrated that yourself by testing the same thing with JVM and it crashing.

Post #5

ollran

I think it is about 500MB because I am able to run JVM with these settings:

$ java -XX:MaxHeapSize=500m -XX:CompressedClassSpaceSize=168m HelloWorld

I am just experimenting with a blank Phoenix application. It must be some of Phoenix’s dependencies that uses asn1.

I am just experimenting with a blank Phoenix app because I’d like to get rid of my Node.js application.

No, it was not the same application. It was a test application that was used to test what happens if JVM runs out of memory. I have no problem with JVM, it works fine. I am experimenting with Elixir because I find it very interesting.

Post #6

jola

Yeah, then I’d say something is up. I’m running a simple Phoenix JSON API in production with a limit of 128MB and it’s using 80MB, never getting close to the limit.

Are there even any nifs shipped with a base phoenix application?

Post #7

NobbZ

Asn.1 is an application bundled with Erlang. In a fresh created project using mix phx.new I can’t find it in the app tree (mix app.tree).

So it has to be included manually.

Post #8

ollran

I tried this:

$ mix phx.new --version
Phoenix v1.4.8
$ mix phx.new test
$ cd test
$ mix app.tree | grep -A 1 -B 8 asn1
│   ├── cowboy
│   │   ├── crypto
│   │   ├── cowlib
│   │   │   └── crypto
│   │   └── ranch
│   │       └── ssl
│   │           ├── crypto
│   │           └── public_key
│   │               ├── asn1
│   │               └── crypto

There seems to be asn1 under the public_key.

Post #9