Testing Only Code and Dead-Code Elimination

[yordisprieto]

Something I really appreciate from our ecosystem is that we trying to include “Test Support” files as part of the dependency packages.

Recently I have had a dilemma,

1. Create two packages (a normal package, and a testing package) and publish them independently, for example: one_piece_commanded and one_piece_commanded_testing.
2. Check for Mix.env() at compile-time and more the code

if Mix.env() == "test" do
  defmodule MyPackage.TestSupport do
    def hello do
      :hello_world
    end
  end
end

The first idea, is way too much work maintaining the packages. Incrementally becoming worst over time the more packages I have to maintain and whatnot.

The second idea, I feel dirty even suggesting it but I am not sure of any other solution that wouldn’t compile into production code that was only meant to be used in testing. Increasing the binary side (ideally, shouldn’t be the case).

Ideally, Elixir would help in the subject matter somehow, not sure what it would look like.

Any thoughts on the topic? How do you deal with the situation?

P.S: Kind of related to the trick of doing if Code.ensure_loaded?(Absinthe) do: .... to load extra code or not based on deps being installed.

[Marcus]

I am not sure if I understand you right. But you can setup compile path per environment.

defmodule MyApp.Mixfile do
  use Mix.Project

  def project do
    [
      ...
      elixirc_paths: elixirc_paths(Mix.env()),
      ...
    ]
  end

  defp elixirc_paths(:test), do: ["lib", "test/support"]
  defp elixirc_paths(_), do: ["lib"]
...

[csadewa]

Ah, is this a discussion on what step elixir ecosystem/community should do to help reduce Testing Package code into production / ensure these Testing Support Package code not compiled in production? in that case i have no real comment

Original post:

Is including Test Support package/module directly on the library without any custom processing a viable option?
`oban` library include testing module (https://hexdocs.pm/oban/Oban.Testing.html#content) directly without any flag and rely on developer to not misuse it for production code

[LostKobrakai]

oban is not the only library doing that. The same applies to Plug.Test, Phoenix.ConnTest or Phoenix.LiveViewTest.

If test helpers are not meant to be bundled in prod code I’d consider a compile time config like config :my_app, compile_test_helpers: false to disable them. Every other toggle will be a heuristic and therefore just waiting to not work for some user.

Generally I’d really consider hard, why including test helpers in prod code would be problematic though. What if the usecase is a test runner and prod actually runs tests?

[yordisprieto]

I went for a little bit of an ugly route: https://github.com/straw-hat-team/beam-monorepo/blob/master/apps/one_piece_commanded/docs/explanations/enhancing-testing-environments.md

I rather do what Oban.Testing is doing, except that it is being compiled for production.

@LostKobrakai

Every other toggle will be a heuristic and therefore just waiting to not work for some user.

That is why I wish Elixir come up with some shared solution since we all doing whatever we think is correct trying to have a solution to the same problem not sure what is the right answer.

@LostKobrakai

What if the usecase is a test runner and prod actually runs tests?

In that case, you would compile the test directory or whatever using elixirc_paths properly. In case of dependency packages, then I wouldn’t consider such code as “testing only”.

@Marcus

I am not sure if I understand you right. But you can setup compile path per environment.

Sorry for the confusion @Marcus I can’t simply do that because the testing code lives in dependency packages, not my package.

[LostKobrakai]

@yordisprieto

That is why I wish Elixir come up with some shared solution since we all doing whatever we think is correct trying to have a solution to the same problem not sure what is the right answer.

Imo the issue here is the assumption that test helpers are code only used in tests. For a library I don’t think that’s an assumption, that should be made and therefore I actually support the existing approaches of having those modules be handled just like any other.

There are some valid concerns like file size bloat, but those are optimizations and also likely most relevant in really constraint environments, where you’d have the same concern not only about modules holding test helpers, but rather any module. So for the file size concern it makes sense to use a wholistic approach of dealing with it, rather than thinking about those handful of modules specific to test helpers.

[yordisprieto]

@LostKobrakai

So for the file size concern it makes sense to use a wholistic approach of dealing with it, rather than thinking about those handful of modules specific to test helpers.

Fair enough,

Just for the sake of exercise to anticipate problems, what would be a good RFC to tackle the problem from Elixir itself?

[yordisprieto]

I keep coming back to this after 2 years, but I am still not sure what is most beneficial for the ecosystem. I wish we would align so we do not need to learn new patterns or intended usage across different packages.

• *.Test*
• *.TestSupport*
• .Testing*
• *Test
• [Insert N way Here]

There are so many different ways to do this. Aside from this, under what configuration key should we put the “feature flagging” (similar to Rust)?

# or alternative just making a point
config :my_scope, features: [test_support: :disabled]

I am not sure; it sounds like a good idea to me, but plenty of libraries would benefit from shipping with even more testing code included: Assertions, Mockings, Factories, Helpers … so many things

[yordisprieto]

@yordisprieto

if Mix.env() == "test" do
  defmodule MyPackage.TestSupport do
    def hello do
      :hello_world
    end
  end
end

To be clear, that wouldn’t work after; Mix.env() inside a dependency for a given app since to always be :prod
That is out of the equation and must be done differently