odarriba

odarriba

Elixir 1.9 release slow boot time

Hello,

We are working in a pretty big Elixir application (at the moment of writing, 1 minute of compilation in an i5 processor and ~930 modules) and we have recently moved from Distillery to new Mix releases.

The issue we are having is that we are experiencing a 300% extra boot time in the application, which is causing some problems on our deployments (but we can fix that, the deployment is not the issue).

We have made a comparision of booting times with a release made by Distillery and a release made by new Mix release command, and we are seeing:

  • Distillery: ~2 seconds until Phoenix is up, inluding checking for pending migrations
  • Mix: ~9 seconds until Phoenix is up, without any migration-related process.

both measurements are on the same machine (which is the same I have made the build, so no cross-platform issue)

I have tries removing all runtime configurations, which make irt a bit better (1 or 2 seconds less) but not even near Distillery times.

Also, I noticed that during Mix release boot there is only one BEAM process taking 150% of CPU for that ~9 seconds, when with Distillery one dont get to that usage (or it goes so faster that top doesn’t measure it).

Our mix.exs release config:

  def project do
    [
      ...
      releases: [
        my_release: [
          include_executables_for: [:unix],
          path: "rel/build/my_release"
        ]
      ],
      default_release: :my_release,
      ...
    ]
  end

vm.args.eex is like the default one and the env.sh.eex just set the distribution to name and sets the node name according to the pod IP when deployed on kubernetes (that was also made on Distillery releases).

Note that we are doing all this benchmarks on local, not even deploying to Kubernetes (which quotas configuratiuons can also affect).

Any idea what is happening?

Most Liked

odarriba

odarriba

Yep, we are using Erlang 22 with Elixir 1.9.0 (we also tried 1.9.1)

In development we are using:

elixir 1.9.0-otp-22
erlang 22.0.5

and in production we build using official elixir image.

However, the compilation times are the expected ones (more or less, it takes time but it’s due to cross dependencies between modules), the issue is when we start the release itself, that produces a CPU spike and a quite long time until the first request is served, which doesn’t happen with Distillery.

tcoopman

tcoopman

Ok, so I’ve created a bunch of test cases and measured the time to entry in Application.start:

## Distillery (elixir 1.8.2-otp-21)
- date && _build/prod/rel/app/bin/app foreground: < 3 seconds
- date && _build/prod/rel/app/bin/app console: < 3 seconds

## Distillery (elixir 1.9.1-otp-21)
- date && _build/prod/rel/app/bin/app foreground: < 3 seconds
- date && _build/prod/rel/app/bin/app console: < 3 seconds

## Elixir releases (elixir 1.9.1-otp-21)
- date && _build/prod/rel/app/bin/app start: 10 seconds
- date && _build/prod/rel/app/bin/app start_iex: 10 seconds

## env MIX_ENV=prod mix phx.server --preload-modules (elixir 1.9.1-otp-21)
- date && env MIX_ENV=prod mix phx.server --preload-modules: < 3 seconds

## Experiment: Elixir releases without runtime configuration (elixir 1.9.1-otp-21)
- date && _build/prod/rel/app/bin/app start: < 3 seconds

As you can see, the Elixir releases with runtime configuration are way slower than distillery (which has the same runtime configuration). By accident I tried releases without runtime configuration and noticed this was much faster (in the same range as distellery - my feeling says it’s still a bit slower, but hard to measure/benchmark correctly so don’t trust me on this)
(Just as confirmation, my runtime configuration does nothing special, only reading environment variables and parsing a 4 values to Integer. It’s the same configuration that I use in distillery as well)

I’ve also tried: env MIX_ENV=prod mix phx.server --preload-modules which should preload and the startup speed is in the same ballpark as distillery, releases without runtime configuration.

So my conclusion is that there is something wrong with elixir releases with runtime configuration.
Maybe that’s easier to confirm for someone else?

odarriba

odarriba

This happens to me also, it’s a private project with a high number of modules, and cannot publish it open source.

However I see the same issue you are having, but in a different scale (because of the number of modules, which is totally understandable).

As @jola said, it is expected that Releases take longer than mix phx.server, the real issue is that mix releases are taking 3x time the start time of a distillery release in some scenarios (we have 4 elixir projects in prod, and only this one is suffering this issue).

In our particular case, we tried removing some aspects like external libraries (AppSignal for example) in case it was causing some trouble starting the agent… but no luck. We end up rolling back to distillery until this gets solved or we have more time to investigate more to fill a proper issue in the core.

if someone thinks about anything we can look at, I will be pleased to try things.

Last Post!

tcoopman

tcoopman

@odarriba maybe you can add your experience with the slow releases to the issue as well. I’ve been able to reproduce an open source example, but the slowness is related to the releases.exs as well.

Where Next?

Popular in Questions Top

nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

Other popular topics Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement