Slow first request on application boot, second is fast

Hey,

I’ve been trying to track down odd behaviour that I’m experiencing with running my Elixir application in Production where by the first request to that specific module/function in my Phoenix controller is slow, and all further requests respond in the expected time.

At first I thought it was an issue with my code around fetching data from ETS, but then figured out with the help of a fellow Alchemist on the Slack channel that it happens even to a raw plug that does no interactions other than passing a short string of JSON exhibits the same behaviour.

It was his idea to attempt to use Distillery to create a release and see if that had any affect, and the odd part is that it does.

The first screenshot is the odd behaviour in question, a simple 5 line plug responding in 28ms.
The second shows how this is not an issue when running my application via Distillery.

Imgur

Had to put the second image on Imgur as I’m a new member :frowning:

3 Likes

This is probably due to code loading. The Erlang code loader supports two modes: interactive and embedded. The former loads BEAM files on-demand, the latter loads all required code for a release upon startup. See http://erlang.org/doc/man/code.html

When you run with mix, interactive mode is used, which causes some modules to be loaded upon the first request. Distillery creates a release, which uses embedded mode.

3 Likes

just curious if you have
build_embedded: Mix.env == :prod,
start_permanent: Mix.env == :prod,

in your mix.exs?

1 Like

I figured the issue was to do with code loading, but wasn’t sure of any fix for this :frowning:

@outlog - I do indeed! They are pretty much defaults for mix projects

2 Likes

Isn’t that the purpose of the embedded flag in mix.exs? These were all run in prod mode.

1 Like

That flag changes the some aspects of the compilation process, perhaps in support of using embedded mode, but it doesn’t actually enable that mode. Calling :code.get_mode() will return :interactive even in prod.

I’m not sure it is possible to use embedded mode outside of a release. Passing --erl "-mode embedded" or ERL_FLAGS="-mode embedded" always result in a crash of the boot process for me.

2 Likes

Aha! That explains a lot then. I suppose that’s why the mix.exs flag is :build_embedded not :start_embedded or something like that.

2 Likes

@cbarratt: as you’ve seen, one fix is to build a release. The alternative would be for you to ‘warm up’ your application yourself, as part of your application initialization or through deployment tooling…

1 Like

This doesn’t seem super viable because each unique code path may load some modules and not others. It isn’t as simple as just hitting a health check endpoint because there’s tons of modules that such a request won’t hit.

1 Like

This is correct. The proper fix to build a release as it contains all the instructions for get your app up and running without any dynamic/interactive code loading.

5 Likes

Thank you both for this.

I could not get “-mode embedded” running today, so I’m really glad I found this thread and these answers.