Incendium - easily profile your web applications with flamegraphs

Incendium

Easy profiling for your Phoenix controller actions (and other functions) using flamegraphs.

Example flamegraph: Example flamegraph — Incendium v0.2.1

Documentation can be found at https://hexdocs.pm/incendium.

Rationale

Profiling Elixir code is easy using the default Erlang tools, such as fprof.

These tools produce a lot of potentially useful data, but visualizing and interpreting all that data is not easy.

The erlang tool eflame contains some utilities to generate flamegraphs from the results of profiling your code

But… eflame expects you to manually run a bash script, which according to my reading seems to call a perl (!) script that generates an interactive SVG.

And although the generated SVGs support some minimal interaction, it’s possible to do better.

When developping a web application, one can take advantage of the web browser as a highly dynamic tool for visualization using SVG, HTML and Javascript.

Fortunately there is a very good javascrtip library to generate flamegraphs: d3-flame-graph.

By reading :eflame stacktrace samples and converting them into a format that d3-flame-graph can understand, we can render the flamegraph in a webpage.

That way, instead of the manual steps above you can just visit an URL in your web application.

Usage

To use incendium in your web application, you need to follow these steps:

1. Add Incendium as a dependency

def deps do
  [
    {:incendium, "~> 0.2.0"}
  ]
end

You can make it a :dev only dependency if you wish, but Incendium will only decorate your functions if you’re in :dev mode. Incendium decorators won’t decorate your functions in :prod (profilers such as eflame should never be used in :prod because they add a very significant overhead; your code will be ~10-12 times slower)

2. Create an Incendium Controller for your application

# lib/my_app_web/controllers/incencdium_controller.ex
defmodule MyApp.IncendiumController do
  use Incendium.Controller,
    routes_module: MyAppWeb.Router.Helpers,
    otp_app: :my_app
end

There is no need to define an accompanying view.

Currently the controller is not extensible (and there aren’t many natural extension points anyway).
Upon compilation, the controller will automcatically add the files incendium.js and incendium.css to your priv/static directory so that those static files will be served using the normal Phoenix mechanisms.
On unusual Phoenix apps which have static files in other places, this might not work as expected.
Currently there isn’t a way to override the place where the static files should be added.

3. Add the controller to your Router

# lib/my_app_web/controllers/router.ex

  require Incendium

  scope "/incendium", MyAppWeb do
    Incendium.routes(IncendiumController)
  end

4. Decorate the functions you want to profile

Incendium decorators depend on the decorator package.

defmodule MyAppWeb.MyContext.ExampleController do
  use MyAppWeb.Mandarin, :controller
  # Activate the incendium decorators
  use Incendium.Decorator

  # Each invocation of the `index/2` function will be traced and profiled.
  @decorate incendium_profile_with_tracing()
  def index(conn, params) do
    resources = MyContext.list_resources(params)
    render(conn, "index.html", resources: resources)
  end
end

Currently incendium only supports tracing profilers (which are very slow and not practical in production).
In the future we may support better options such as sampling profilers.

5. Visit the /incendium route to see the generated flamegraph

Each time you run a profiled function, a new stacktrace will be generated. Stacktraces are not currently saves, you can only access the latest one. In the future we might add a persistence layer that stores a number of stacktraces instead of keeping just the last one.

Here you can find an example flamegraph with explanations about how to interact with it .

25 Likes

Some questions “for the audience”…

  1. Should I add a way of saving profiler stacktraces for multiple function runs and aggregate them all in the same flamegraph?

  2. Should I add a way to compare flamegraphs between different implementations of the same function? How should I do it? I can’t find a good way of comparing flamegraphs…

  3. Any suggestions for a good sampling profilers to use? (as opposed to a “tracing” profiler)

This is nice :slight_smile: But how could it be used if an app does not use phoenix framework ?

Well, you can use the private API to generate HTML files and dump them somewhere.

Or maybe I could change things so that Incendium was a standalone phoenix app locally (listening to a different port, of course). That would make it independent of the user’s app, and it would allow you to profile any Elixir code running locally.

That’s probably the easiest way

PS. This was written as a very simple way of profiling routes in a Phoenix app, and although I’ve tried to make it generic and usable from other apps, I’ve always thought of making it a phoenix controller instead of a standalone phoenix app

Incendium can now (v0.3.0) integrate with Benchee to generate reproducible benchmarks with performance data displayed as flamegraphs (using samples from multiple iterations, of course).

The logical way would be to create an Incendium formatter for benchee, but unfortunately that’s impossible. Incendium needs to run the benchmark twice (one without profiling so that we get accurate runtimes and another with profiling so that we get flamegraphs). This means I need control before execution (to set up profiling and create some processes that will gather the relevant metrics generated by the profiler), during execution (so that I can run the benchmark suite tiwce) and after execution (as a normal benchee formatter, to generate the HTML report

The limitations above mean that you need to call Incendium.run(benchmark, options) instead of Benchee.run(benchmark, options_with_incendim_specific_parts).

Example HTML report here: https://hexdocs.pm/incendium/0.3.0/assets/Example.html
Note that the flamegraph width is scaled according to the scenario runtime (longer runtimes create wider flamegraphs). The scaling of flamegraph widths can be disabled with an option.

Pinging @PragTob because he might be interested in this. I’d appreciate a way of running this with Benchee.run/2 instead of having to call Incendum.run/2.

The old Pheonix-based features still work as before.

2 Likes

:wave:

Thanks for the ping.

Sad things first, I’ve been battling hand/arm problems for close to 5 months and haven’t done any OSS or gaming or general fun things in the time, hence benchee might appear stale… and there’s also a lot of unreleased stuff on main from even before that :’(

We should definitely have that, and we might already have it!

There is a feature that has been on main forever (thanks to idea from @josevalim and implementation by @pablocostass ) to have profile_after which runs builtin profilers after benchmarks initial PR

It should be semi easy to extend it to support non builtin profilers/that you can give it custom args. Maybe it already does, can’t check right now.

That said, I worry about my hands and if the past is an indicator of the future won’t get to make a new release or changes any time soon. Sorry :cry: It’s scary for me so I’m minimizing what I can to have a chance to be “good” again. Once I am, fixing that damn mac bug in nano time measurements releasing a new versiona nd brushing up benchee (and simplecov) is top of the priority list.

3 Likes

Oh, hope you get better soon!

Does the profiler have access to the results of the original suite so that I can report the original runtimes and do some sanity checks on whether the profilers have distorted the relationship between the runtimes of different scenarios?

Profile is run last - it has access to everything. However, a quick look at Profile shows that we are only calling the profilers themselves with the functions. So, some further work would be needed there.

That said, one of the easiest things might be for you for now to do:

Benche.run(..)
|> YourModule.your_call

Benchee funs always return the suite which has all the aggregated information throughout the benchmark run.

If calling the profiler on the suite is all I need, then I don’t think I’ll need anything else

Another question: do you think that scaling the flamegraph widths should be the default, so that longer runtimes result in wider flamegraphs? Or should all the flamegraphs have the same width by default?

This will be configurable, of course, I’m just asking you what the default should be

to me scaling by default sounds good, to a level where I want to be able to see and comprehend it. Picking the scaling factor etc. is the real challenge :slight_smile:

I’m scaling them proportionally to the runtime. The longest running scenario gets full width and the rest get partial width in proportion to how much faster they are

Hi,

I used this tool on my app and already got some learnings, thanks !

Incendium decorators won’t decorate your functions in :prod

I’like to do profiling on an instance of my app as close as possible as my prod. Is there any way to build the app as prod and force Incendium to trace the functions I’ve decorated ?

Many thanks

I think there is no such thing as “tracing the functions” as close as possible as prod. The profiler used by Incendium instruments your code so much that it would no longer be “close to prod”… At least in terms of efficiency