Creating a production release with :cover information

If you try this out I’d love to read about how it goes!

And of course I forgot to add a question…

What would be the steps to build such a release with such info? Has anyone done anything like this and willing to share their experience?

Had some time to look into it and got coverage info in a production release. Was able to run coverage analysis, however failed at getting a nicely formatted coverage report files because source code for modules is missing in the release.

Not sure how to include source code in the final release. Maybe someone knows? Until then, I’ll
document everything I did so far. Here are the steps:

1. Add :tools: to a list of extra_applications in mix.exs:

       def application do
         [
   -       extra_applications: [:logger],
   +       extra_applications: [:logger, :tools],
           mod: {MyApp, []}
         ]
   

2. Specify a strip_beams option in the releases section of mix.exs:

            include_executables_for: [:unix],
            applications: [
              myapp: :permanent
   +        ],
   +        strip_beams: [
   +          keep: ["Dbgi"]
            ]
          ]
   
   

   The strip_beams option is documented here.

   Keeping just the Dbgi chunk appears to be sufficient for :cover to work, but the full list of so-called chunkIDs can be found here.

3. Compile, then start the release and connect to a remote console:

   /app/bin/myapp remote
   

4. Execute this line:

   :cover.compile_beam_directory(~c"/app/lib/myapp-1.0.0/ebin")
   

   Here, /app/lib/myapp-1.0.0/ebin is a path where ebin files for a release are stored. This line is inspired by the following snippet from the Elixir source code: link.

   For the command to finish it took a good several seconds on my M3-based MacBook to process ~350 modules.

5. That’s it, the modules are now :cover-compiled. But you can confirm this once again by running this:

   :code.which(MyApp.MyModule)
   # => :cover_compiled
   

6. At this point, I can make the application do some work, like serve HTTP requests during execution of an e2e tests suite.

7. Finally, see the results by connecting to the release remotely and running :cover.analyse(). The output will looks
   something like this:

   {:result, [
     {{MyApp.Web.StoredContractController, :__info__, 1}, {0, 9}},
     {{MyApp.Web.StoredContractController, :action, 2}, {1, 0}},
     {{MyApp.Web.StoredContractController, :"action (overridable 2)", 2},
      {1, 0}},
     {{MyApp.Web.StoredContractController, :call, 2}, {1, 0}},
     {{MyApp.Web.StoredContractController, :check_stored_contract_limit, 2},
      {1, 1}},
     {{MyApp.Web.StoredContractController, :contract_to_pdf, 1}, {0, 7}},
     {{MyApp.Web.StoredContractController, :create, 2}, {11, 2}},
     {{MyApp.Web.StoredContractController, :delete, 2}, {0, 5}},
     {{MyApp.Web.StoredContractController, :download_pdf, 2}, {0, 9}},
     {{MyApp.Web.StoredContractController, :fetch_or_create_owner_account,
       2}, {1, 1}},
     {{MyApp.Web.StoredContractController, :index, 2}, {0, 2}},
     {{MyApp.Web.StoredContractController, :init, 1}, {1, 0}},
     {{MyApp.Web.StoredContractController,
       :load_stored_contract_for_account, 2}, {5, 2}},
     {{MyApp.Web.StoredContractController, :load_stored_contract_for_owner,
       2}, {0, 3}},
     {{MyApp.Web.StoredContractController, :move, 2}, {0, 5}},
     {{MyApp.Web.StoredContractController, :phoenix_controller_pipeline, 2},
      {1, 0}},
     {{MyApp.Web.StoredContractController, :preview, 2}, {0, 1}},
     {{MyApp.Web.StoredContractController, :show, 2}, {1, 0}},
     {{MyApp.Web.StoredContractController, :stored_contract_id, 1}, {1, 0}},
     {{MyApp.Web.StoredContractController, :update, 2}, {0, 5}},
     {{MyApp.Web.StoredContractController, :update_groups, 2}, {0, 6}},
     {{MyApp.Web.StoredContractController, :update_reminders, 2}, {0, 7}},
     {{MyApp.User.Account.Authenticator, :__info__, 1}, {0, 9}},
     {{MyApp.User.Account.Authenticator, :assign_password_hash, 1},
      {0, 2}},
     {{MyApp.User.Account.Authenticator, :password_matches_hash?, 2},
      {0, 2}},
     {{MyApp.User.Account.Authenticator,
       :sign_in_token_matches_hash?, 2}, {0, 2}},
     {{MyApp.HTTPClient.MySMTP, :__info__, 1}, {0, 9}},
     {{MyApp.HTTPClient.MySMTP, :fix_response, 1}, {0, 1}},
     {{MyApp.HTTPClient.MySMTP, :get_log, 1}, {0, 1}},
     {{MyApp.HTTPClient.MySMTP, :login, 0}, {0, 1}},
     {{MyApp.HTTPClient.MySMTP, :new, 0}, {0, 1}},
     {{MyApp.Query, :__info__, 1}, {0, 9}},
     {{MyApp.Query, :not_deleted, 1}, {1, 0}},
     {{MyApp.Web.PartyView, :__info__, 1}, {0, 9}},
     {{MyApp.Web.PartyView, :__resource__, 0}, {1, 0}},
     {{MyApp.Web.PartyView, :do_render, 5}, {5, 0}},
     {{MyApp.Web.PartyView, :opened_at, 2}, {4, 2}},
     {{MyApp.Web.PartyView, :render, 2}, {2, 1}},
     {{MyApp.Web.Admin.EnsureAuthPlug, :__info__, 1}, {0, 9}},
     {{MyApp.Web.Admin.EnsureAuthPlug, :call, 2}, {0, 3}},
     {{MyApp.Web.Admin.EnsureAuthPlug, :init, 1}, {0, 1}},
     {{MyApp.Reminder.UseCase, :__info__, 1}, {0, 9}},
     {{MyApp.Reminder.UseCase, :build_document_payload, 1}, {0, 12}},
     {{MyApp.Reminder.UseCase, :mark_done, 2}, {0, 2}},
     {{MyApp.Reminder.UseCase, :send_notifications, ...}, {0, ...}},
     {{MyApp.Reminder.UseCase, ...}, {...}},
     {{...}, ...},
     {...},
     ...
   ], []}
   

Notes:

• Sadly, as I mentioned, I couldn’t make saving the analysis to an easily digestable file format work. That is, attempting to run :cover.analyse_to_file() will produce output that looks like this:

  {:result, [],
   [
     no_source_code_found: Jason.Encoder.Range,
     no_source_code_found: Jason.Encoder.Stream,
     no_source_code_found: MyApp.Module1,
     no_source_code_found: MyApp.Module2,
     # ...
  ]}
  

  The documentation for :cover.analyse_to_file/2 mentions locations where the function will look for source code. But I couldn’t quickly find how to make mix compile or mix release include the source code of the application together with the release. Bluntly copying source code files into one of the folders where :cover expects it likely won’t work, since the source code is written in Elixir, while :cover expects source code in .erl? I don’t know and haven’t tried this.

• “Performance” section on “cover - The Coverage Analysis Tool” page in Erlang documentation has this to say:

  Execution of code in cover-compiled modules is slower and more memory consuming than for regularly compiled modules. As the Cover database contains information about each executable line in each cover-compiled module, performance decreases proportionally to the size and number of the cover-compiled modules.

  It’s an important note and I haven’t had the chance to do more tests & comparisons to understand exactly how slower / resource-hungry the app becomes.

Figured out how to generate the .html reports:

1. While connected to a release remotely, generate export:

   :cover.export(~c"/tmp/export.coverdata")
   

2. Copy the export file to the root of application source code (e.g. the to folder that contains mix.exs, mix.lock, etc.),

3. Prepare a folder to save .html reports into. Run:

   mkdir /tmp/my-cover-info
   

4. Run iex -S mix, then run the following commands:

   :cover.import(~c"export.coverdata")
   :cover.compile_beam_directory(~c"_build/dev/lib/myapp/ebin")
   :cover.analyse_to_file([:html, {:outdir, ~c"/tmp/my-cover-info"}])
   

The reason :cover.analyse_to_file/1 works this time is because it’s able to locate source code by calling MyModuleName.module_info(:compile) (as described in documentation for :cover.analyse_to_file/1.

Sadly, the contents inside .html files look like this:

CleanShot 2024-10-16 at 15.41.58@2x826×300 21.3 KB

E.g. a weird :-( symbol and no green lines whatsoever, as well as completely missing execution counter info. So, I must be doing something wrong still

Figured what the problem was - need to compile the modules first, only then import the :cover data. Here’s the correct sequence to produce the .html report:

:cover.compile_beam_directory(~c"_build/dev/lib/myapp/ebin")
:cover.import(~c"export.coverdata")
:cover.analyse_to_file([:html, {:outdir, ~c"/tmp/my-cover-info"}])

As an example, calling functions CoverTest.one(), CoverTest.two() and CoverTest.three() produce the following output - e.g. IO.puts("Hello, world!") was executed 3 times:

CleanShot 2024-10-16 at 22.46.05@2x1958×1124 92.8 KB

Great write up, I’ll be giving this a try