Hello
I’m opening this thread to ask some questions about testing practices when dealing with values set at compile time.
I’m working with Elixir 1.4.1 and ExUnit.
What
The context is a feature flagging library I’m working on (work in progress on GitHub). It’s still very much a “learning project”, so bear with me if some of my doubts are naive.
I’m finding testing the configuration of the library a bit difficult, and I’m looking for suggestions and best practices.
I’ve already read the Plataformatec blogpost on providing client mocks in the test environment, and my doubts are very similar to what discussed in this thread.
My main use case is that I want to let people including the library in their Mix projects to configure it via Mix. The way I understand it, there is no point in including a config/config.exs
file in the Hex package because it won’t be loaded by the host application, so I’m providing the defaults manually and wrapping the config calls in a module:
defmodule FunWithFlags.Config do
@default_redis_config [host: 'localhost', port: 6379]
@default_cache_config [enabled: true]
def redis_config do
case Application.get_env(:fun_with_flags, :redis, []) do
uri when is_binary(uri) ->
uri
opts when is_list(opts) ->
Keyword.merge(@default_redis_config, opts)
end
end
def cache? do
Keyword.get(ets_cache_config(), :enabled)
end
defp ets_cache_config do
Keyword.merge(
@default_cache_config,
Application.get_env(:fun_with_flags, :cache, [])
)
end
end
The good
That works, and I’m quite happy with Config.redis_config/0
reading from Application.get_env
with each call because that function is only invoked when starting a GenServer.
Also, since it’s loaded dynamically at runtime, I can easily test that my function works with this helper:
defp configure_redis_with(conf) do
Mix.Config.persist(fun_with_flags: [redis: conf])
assert ^conf = Application.get_env(:fun_with_flags, :redis)
end
The bad
What I’m less happy with is Config.cache?/0
. The library should be able to operate with or without an intermediate caching layer between its public API and Redis, and I would like to have integration tests for both operation modes. I already have those tests, but the option to disable the cache is new, and they will only test the configuration variant I set manually before running mix test
.
Long story short, that config value is not something I can pass to a start_link()
and then forget about, but it changes the behaviour of a lot of functions invoked directly “in-process” in the host application code.
So far I’ve found two main ways to use it.
1st approach: many runtime calls and module attributes
The first naive one is to query it each time I need it. For example:
def lookup(flag_name) do
do_lookup(flag_name, with_cache: Config.cache?)
end
I don’t expect users to enable or disable the cache layer at runtime without a restart/rebuild, though, especially because it would have to be configured via Mix. This approach doesn’t look very efficient.
If I were to do this, however, I think I would try to store the configuration in module attributes for performance1, and my Config
module would look like this:
defmodule FunWithFlags.Config do
@default_cache_config [enabled: true]
@compiled_ets_cache_config Keyword.merge(
@default_cache_config,
Application.get_env(:fun_with_flags, :cache, [])
)
@is_cache_enabled Keyword.get(@compiled_ets_cache_config, :enabled)
def cache? do
@is_cache_enabled
end
end
This works, but I am not sure how to test it. The @attributes
are defined at compile time, and while all the tests pass (or fail predictably) with the cache enabled and disabled, I have no way to change the configuration during the test run.
Since the first attempt is not viable, let’s see my second approach.
2nd approach: code generation
The second way I can use that config is to choose code dynamically on compilation. A simplified example:
defmodule FunWithFlags.Store do
alias FunWithFlags.Config
if Config.cache? do
def lookup(flag_name) do
# check the cache, then redis, maybe update the cache
end
else
def lookup(flag_name) do
# just check redis
end
end
end
This code works. It works with both the compile-time @attributes
implementation and the dynamic Application.get_env
one. However, I still can’t test it properly!
While I like this solution more than the previous one, it suffers from the same problems when it comes to changing the configuration during the tests.
How can I test both versions, with cache and without?
Other techniques
I guess I could implement the functionality in a number of ways.
The one that would make testing easy, that is, dynamic config lookup and dynamic cache/nocache conditionals, also sounds unnecessarily inefficient at runtime.
I could also have two different FunWithFlags.Store
modules, each implementing an operation mode, and select them based on the config. And yet, again, if I decide to freeze anything at compile time I lose the ability to test.
It’s like the Elixir version of Heisenberg’s uncertainty principle.
For the tests, I could adopt the technique showed in that Plataformatec article and inject a mock in the tests, but I’m not too convinced, because in my case this is not something isolated to a point where I can push it into an edge service or adapter.
Also, then, not only would I still have no way to control the actual application behaviour with the two configurations, but since the mock would be at the core of the library I wouldn’t have any integration test at all.
I know that they’re not recommended, but it sounds like a good use case for Mock:
test "it works without cache" do
with_mock(FunWithFlags.Config, [cache?: fn() -> false end] do
# my integration tests
end
end
Although it doesn’t help with the other problem I haven’t looked at yet: not adding the cache GenServer to the supervision tree is the config is disabled.
Compromises?
At the moment I think that if I want to properly test the library, I can’t use the optimization techniques I thought of. Or, if decide to use them, I must accept to not have proper automated tests: I can still change the configuration manually and run the test suite two times. I guess I could automate this with a shell script, but is there a builtin Elixir way to do this?
In a dynamic and interpreted language I would know what to reach for, but I’m not too familiar with Elixir and OTP yet, and I’m not sure what the proper way to deal with this would be.
Recompiling a reloading a module would work, I guess.
Am I missing something? Is there an established way to tackle this problem?
Thank you!
1 - IIRC Application.get_env
reads from an ETS table, but that would still be a lot of frequent ETS reads. Please correct me if I shouldn’t worry about this.