Testing a Mix app with ExUnit: changing config values set at compile time

Hello

I’m opening this thread to ask some questions about testing practices when dealing with values set at compile time.
I’m working with Elixir 1.4.1 and ExUnit.

What

The context is a feature flagging library I’m working on (work in progress on GitHub). It’s still very much a “learning project”, so bear with me if some of my doubts are naive.

I’m finding testing the configuration of the library a bit difficult, and I’m looking for suggestions and best practices.

I’ve already read the Plataformatec blogpost on providing client mocks in the test environment, and my doubts are very similar to what discussed in this thread.

My main use case is that I want to let people including the library in their Mix projects to configure it via Mix. The way I understand it, there is no point in including a config/config.exs file in the Hex package because it won’t be loaded by the host application, so I’m providing the defaults manually and wrapping the config calls in a module:

My fun_with_flags/config.ex:

defmodule FunWithFlags.Config do
  @default_redis_config [host: 'localhost', port: 6379]
  @default_cache_config [enabled: true]

  def redis_config do
    case Application.get_env(:fun_with_flags, :redis, []) do
      uri when is_binary(uri) ->
        uri
      opts when is_list(opts) ->
        Keyword.merge(@default_redis_config, opts)
    end
  end

  def cache? do
    Keyword.get(ets_cache_config(), :enabled)
  end

  defp ets_cache_config do
    Keyword.merge(
      @default_cache_config,
      Application.get_env(:fun_with_flags, :cache, [])
    )
  end
end

The good

That works, and I’m quite happy with Config.redis_config/0 reading from Application.get_env with each call because that function is only invoked when starting a GenServer.

Also, since it’s loaded dynamically at runtime, I can easily test that my function works with this helper:

defp configure_redis_with(conf) do
  Mix.Config.persist(fun_with_flags: [redis: conf])
  assert ^conf = Application.get_env(:fun_with_flags, :redis)
end

The bad

What I’m less happy with is Config.cache?/0. The library should be able to operate with or without an intermediate caching layer between its public API and Redis, and I would like to have integration tests for both operation modes. I already have those tests, but the option to disable the cache is new, and they will only test the configuration variant I set manually before running mix test.

Long story short, that config value is not something I can pass to a start_link() and then forget about, but it changes the behaviour of a lot of functions invoked directly “in-process” in the host application code.

So far I’ve found two main ways to use it.

1st approach: many runtime calls and module attributes

The first naive one is to query it each time I need it. For example:

def lookup(flag_name) do
  do_lookup(flag_name, with_cache: Config.cache?)
end

I don’t expect users to enable or disable the cache layer at runtime without a restart/rebuild, though, especially because it would have to be configured via Mix. This approach doesn’t look very efficient.
If I were to do this, however, I think I would try to store the configuration in module attributes for performance1, and my Config module would look like this:

defmodule FunWithFlags.Config do
  @default_cache_config [enabled: true]

  @compiled_ets_cache_config Keyword.merge(
    @default_cache_config,
    Application.get_env(:fun_with_flags, :cache, [])
  )
  @is_cache_enabled Keyword.get(@compiled_ets_cache_config, :enabled)

  def cache? do
    @is_cache_enabled
  end
end

This works, but I am not sure how to test it. The @attributes are defined at compile time, and while all the tests pass (or fail predictably) with the cache enabled and disabled, I have no way to change the configuration during the test run.

Since the first attempt is not viable, let’s see my second approach.

2nd approach: code generation

The second way I can use that config is to choose code dynamically on compilation. A simplified example:

defmodule FunWithFlags.Store do
  alias FunWithFlags.Config

  if Config.cache? do
    def lookup(flag_name) do
      # check the cache, then redis, maybe update the cache
    end
  else
    def lookup(flag_name) do
      # just check redis
    end
  end
end

This code works. It works with both the compile-time @attributes implementation and the dynamic Application.get_env one. However, I still can’t test it properly!
While I like this solution more than the previous one, it suffers from the same problems when it comes to changing the configuration during the tests.

How can I test both versions, with cache and without?

Other techniques

I guess I could implement the functionality in a number of ways.
The one that would make testing easy, that is, dynamic config lookup and dynamic cache/nocache conditionals, also sounds unnecessarily inefficient at runtime.

I could also have two different FunWithFlags.Store modules, each implementing an operation mode, and select them based on the config. And yet, again, if I decide to freeze anything at compile time I lose the ability to test.

It’s like the Elixir version of Heisenberg’s uncertainty principle. :slight_smile:

For the tests, I could adopt the technique showed in that Plataformatec article and inject a mock in the tests, but I’m not too convinced, because in my case this is not something isolated to a point where I can push it into an edge service or adapter.

Also, then, not only would I still have no way to control the actual application behaviour with the two configurations, but since the mock would be at the core of the library I wouldn’t have any integration test at all.

I know that they’re not recommended, but it sounds like a good use case for Mock:

test "it works without cache" do
  with_mock(FunWithFlags.Config, [cache?: fn() -> false end] do
     # my integration tests
  end
end

Although it doesn’t help with the other problem I haven’t looked at yet: not adding the cache GenServer to the supervision tree is the config is disabled.

Compromises?

At the moment I think that if I want to properly test the library, I can’t use the optimization techniques I thought of. Or, if decide to use them, I must accept to not have proper automated tests: I can still change the configuration manually and run the test suite two times. I guess I could automate this with a shell script, but is there a builtin Elixir way to do this?

In a dynamic and interpreted language I would know what to reach for, but I’m not too familiar with Elixir and OTP yet, and I’m not sure what the proper way to deal with this would be.

Recompiling a reloading a module would work, I guess.

Am I missing something? Is there an established way to tackle this problem?

Thank you!


1 - IIRC Application.get_env reads from an ETS table, but that would still be a lot of frequent ETS reads. Please correct me if I shouldn’t worry about this.

2 Likes

Sure they can, and in fact some of the ones I’ve made and use at work have just such a config/defaults.exs, it is mostly for documentation since access by code directly sets defaults, but it is still convenient to have and to make helper functions/macros inside of. :slight_smile:

Near the top of my config/config.exs in my root project I have a few lines of:

import_config "../deps/*/config/defaults.exs"
import_config "../deps/*/config/#{Mix.env}_defaults.exs"

That way any dependency that follows the same pattern gets its defaults loaded as well. :slight_smile:

It is still best to set defaults in-code though as always (unless a default does not make sense and the user should definitely set it).

The Application.*env* functions are fast, like really fast, do not worry about accessing it being slow. Configs stored dynamically in ETS are also blazing fast, you’d be surprised just how fast ETS is, feel free to access it on-demand.

For a Caching layer though I’d make it a configured process PID or name, depending on what process is started there it either caches or it does not, it can even be swapped out in real-time without anything else in the system knowing or caring.

For the code generation style you can test it, but you need to make more than one testing environment and be sure to test using them all (via alias’s for example).

Another nice thing about if you were to make a dedicated process for connecting and it was either caching or not (and could be swapped at runtime) is that mocking it would then become dead-simple. :slight_smile:

Just remember, if in doubt about a design, you probably need to break it up into more actors/processes, they are fast and they scale.

Absolutely not, ETS is fast, designed to be fast. :slight_smile:

3 Likes

Thanks for the reply

So, the library’s config.exs is loaded only if the developers of host applications import it correctly. I imagined this was possible, but it doesn’t seem very reliable. Still, good to know!

Ok, thanks. Good to know :slight_smile:

I’m not sure what do you mean with this.

At the moment the cache is an ETS table created with [:set, :protected, :named_table, {:read_concurrency, true}]. It’s wrapped in a supervised GenServer started with the application.

Writes to the cache are go through casts to the owner process, so that they’re serialized (they’re going to be infrequent or, at best, in short bursts).
Reads, on the other hand, are executed in the invoking process, to minimize the data transferred between processes. I’m vaguely aware that an ETS table is actually a process itself, but I’m trying to support high levels of read concurrency and I’ve read that this approach is better than reading through GenServer.call(table_owner_pid, {:please_read, :my_key}).

Is this what you’re referring to? Or are you suggesting another process that either proxies to the cache or goes directly to Redis? And then I would swap this process if needed?

Ok, that’s interesting. I can think how I would do it by setting MIX_ENV in the terminal. Can you give me an example of how you’d do this with alias please?

1 Like

This. :slight_smile:
Do not worry about spawning more long-lived processes, or even short-lived ones, the system is designed for it. :slight_smile:

I don’t have anything from my own code but you can easily add a mix alias that changes the environment and calls a mix command, then put it and the main ‘test’ environment behind a ‘test’ alias so both get called when you mix test. :slight_smile:

1 Like

I’ve finally had time to give it another go.

Thank you for your help @OvermindDL1, your solution worked really well.
I ended up using a Mix alias and also using different (unit tested) modules that are selected based on the configuration. The integration tests are then repeated with both configurations.

I’m going to summarize here the setup I’ve come up with, in case other people will find it helpful.
I’m leaving references to my specific use case in the code snippets to make it easier to follow their purpose.

###1) How to select different behavior at compile time

The cleanest solution I could find was to implement two modules with the same interface but with different implementations, so I ended up with FunWithFlags.Store that knows about both Redis and the ETS cache, and FunWithFlags.SimpleStore that only uses Redis directly. Both modules are unit tested independently.

Then, I select them at compile time with module attributes. For example:

  defmodule FunWithFlags.Config do
    def cache? do
      Application.get_env(:fun_with_flags, :cache, [enabled: true])[:enabled]
    end
+   def store_module do
+     if __MODULE__.cache? do
+       FunWithFlags.Store
+     else
+       FunWithFlags.SimpleStore
+     end
+   end
  end

  defmodule FunWithFlags do
-   alias FunWithFlags.Store
+   @store FunWithFlags.Config.store_module

    def do_something do
-     Store.do_something
+     @store.do_something
    end
  end

2) Define the configuration

While I provide a default value in the code, for this to work I used the default Mix configuration files. I’m only interested in the test environments, so I’m not bothering to load files for the dev env.

The main config file:

# config/config.exs

use Mix.Config

case Mix.env do
  :test          -> import_config "test.exs"
  :test_no_cache -> import_config "test_no_cache.exs"
  _              -> nil
end

The test one:

# config/test.exs

use Mix.Config

IO.puts "Loading config for default test env"

config :fun_with_flags, :cache,
  enabled: true

And the one that disables the cache:

# config/test_no_cache.exs

use Mix.Config

IO.puts "Loading config for the \"no cache\" integration test env"

config :fun_with_flags, :cache,
  enabled: false

3) Setup a Mix task alias

As documented here, it’s possible to define new Mix tasks that will be available only in this specific project without “bubbling up” to the host applications. I couldn’t really get a mix test override to work, but I actually realized that I find mix test.all more explicit and useful.

defmodule FunWithFlags.Mixfile do
  use Mix.Project

  def project do
    [
      aliases: aliases(),
      # ...
    ]
  end

  defp aliases do
    [
      {:"test.all", [&run_tests/1, &run_integration_tests/1]}
    ]
  end

  defp run_tests(_) do
    Mix.shell.cmd(
      "mix test --color", 
      env: [{"MIX_ENV", "test"}]
    )
  end

  defp run_integration_tests(_) do
    IO.puts "\nRunning the integration tests with the cache disabled."
    Mix.shell.cmd(
      "mix test test/fun_with_flags_test.exs --color",
      env: [{"MIX_ENV", "test_no_cache"}]
    )
  end
end

With this configuration, mix test is still available and the new alias acts as a simple wrapper. I also had to explicitly pass the --color flag because the output was using the default shell foreground color. Depending on what you’re used to, you might want to add more flags.

With this in place, I can run the new alias task and get the two test runs with both configurations. Mix will take care of recompiling the app for the second environment as appropriate.

$ rm -r _build
$ mix test.all
Loading config for default test env
==> connection
Compiling 1 file (.ex)
Generated connection app
==> redix
Compiling 9 files (.ex)
Generated redix app
==> fun_with_flags
Compiling 9 files (.ex)
Generated fun_with_flags app
Running the tests with Mix.env: test
...............................................

Finished in 0.1 seconds
47 tests, 0 failures

Randomized with seed 350185

Running the integration tests with the cache disabled.
Loading config for the "no cache" integration test env
==> connection
Compiling 1 file (.ex)
Generated connection app
==> redix
Compiling 9 files (.ex)
Generated redix app
==> fun_with_flags
Compiling 9 files (.ex)
Generated fun_with_flags app
Running the tests with Mix.env: test_no_cache
.........

Finished in 0.1 seconds
9 tests, 0 failures

Randomized with seed 153788

4) Setup Travis

The last bit is to use the same dual env system on Travis. This is as easy as configuring the script command in the travis.yml file:

  language: elixir
  elixir:
    - 1.4
  otp_release:
    - 19.2
  services:
    - redis-server
+ script: mix test.all
6 Likes