I’ve finally had time to give it another go.

Thank you for your help @OvermindDL1, your solution worked really well.
I ended up using a Mix alias and also using different (unit tested) modules that are selected based on the configuration. The integration tests are then repeated with both configurations.

I’m going to summarize here the setup I’ve come up with, in case other people will find it helpful.
I’m leaving references to my specific use case in the code snippets to make it easier to follow their purpose.

###1) How to select different behavior at compile time

The cleanest solution I could find was to implement two modules with the same interface but with different implementations, so I ended up with FunWithFlags.Store that knows about both Redis and the ETS cache, and FunWithFlags.SimpleStore that only uses Redis directly. Both modules are unit tested independently.

Then, I select them at compile time with module attributes. For example:

  defmodule FunWithFlags.Config do
    def cache? do
      Application.get_env(:fun_with_flags, :cache, [enabled: true])[:enabled]
    end
+   def store_module do
+     if __MODULE__.cache? do
+       FunWithFlags.Store
+     else
+       FunWithFlags.SimpleStore
+     end
+   end
  end

  defmodule FunWithFlags do
-   alias FunWithFlags.Store
+   @store FunWithFlags.Config.store_module

    def do_something do
-     Store.do_something
+     @store.do_something
    end
  end

2) Define the configuration

While I provide a default value in the code, for this to work I used the default Mix configuration files. I’m only interested in the test environments, so I’m not bothering to load files for the dev env.

The main config file:

# config/config.exs

use Mix.Config

case Mix.env do
  :test          -> import_config "test.exs"
  :test_no_cache -> import_config "test_no_cache.exs"
  _              -> nil
end

The test one:

# config/test.exs

use Mix.Config

IO.puts "Loading config for default test env"

config :fun_with_flags, :cache,
  enabled: true

And the one that disables the cache:

# config/test_no_cache.exs

use Mix.Config

IO.puts "Loading config for the \"no cache\" integration test env"

config :fun_with_flags, :cache,
  enabled: false

3) Setup a Mix task alias

As documented here, it’s possible to define new Mix tasks that will be available only in this specific project without “bubbling up” to the host applications. I couldn’t really get a mix test override to work, but I actually realized that I find mix test.all more explicit and useful.

defmodule FunWithFlags.Mixfile do
  use Mix.Project

  def project do
    [
      aliases: aliases(),
      # ...
    ]
  end

  defp aliases do
    [
      {:"test.all", [&run_tests/1, &run_integration_tests/1]}
    ]
  end

  defp run_tests(_) do
    Mix.shell.cmd(
      "mix test --color", 
      env: [{"MIX_ENV", "test"}]
    )
  end

  defp run_integration_tests(_) do
    IO.puts "\nRunning the integration tests with the cache disabled."
    Mix.shell.cmd(
      "mix test test/fun_with_flags_test.exs --color",
      env: [{"MIX_ENV", "test_no_cache"}]
    )
  end
end

With this configuration, mix test is still available and the new alias acts as a simple wrapper. I also had to explicitly pass the --color flag because the output was using the default shell foreground color. Depending on what you’re used to, you might want to add more flags.

With this in place, I can run the new alias task and get the two test runs with both configurations. Mix will take care of recompiling the app for the second environment as appropriate.

$ rm -r _build
$ mix test.all
Loading config for default test env
==> connection
Compiling 1 file (.ex)
Generated connection app
==> redix
Compiling 9 files (.ex)
Generated redix app
==> fun_with_flags
Compiling 9 files (.ex)
Generated fun_with_flags app
Running the tests with Mix.env: test
...............................................

Finished in 0.1 seconds
47 tests, 0 failures

Randomized with seed 350185

Running the integration tests with the cache disabled.
Loading config for the "no cache" integration test env
==> connection
Compiling 1 file (.ex)
Generated connection app
==> redix
Compiling 9 files (.ex)
Generated redix app
==> fun_with_flags
Compiling 9 files (.ex)
Generated fun_with_flags app
Running the tests with Mix.env: test_no_cache
.........

Finished in 0.1 seconds
9 tests, 0 failures

Randomized with seed 153788

4) Setup Travis

The last bit is to use the same dual env system on Travis. This is as easy as configuring the script command in the travis.yml file:

  language: elixir
  elixir:
    - 1.4
  otp_release:
    - 19.2
  services:
    - redis-server
+ script: mix test.all

Sure they can, and in fact some of the ones I’ve made and use at work have just such a config/defaults.exs, it is mostly for documentation since access by code directly sets defaults, but it is still convenient to have and to make helper functions/macros inside of.

Near the top of my config/config.exs in my root project I have a few lines of:

import_config "../deps/*/config/defaults.exs"
import_config "../deps/*/config/#{Mix.env}_defaults.exs"

That way any dependency that follows the same pattern gets its defaults loaded as well.

It is still best to set defaults in-code though as always (unless a default does not make sense and the user should definitely set it).

The Application.*env* functions are fast, like really fast, do not worry about accessing it being slow. Configs stored dynamically in ETS are also blazing fast, you’d be surprised just how fast ETS is, feel free to access it on-demand.

For a Caching layer though I’d make it a configured process PID or name, depending on what process is started there it either caches or it does not, it can even be swapped out in real-time without anything else in the system knowing or caring.

For the code generation style you can test it, but you need to make more than one testing environment and be sure to test using them all (via alias’s for example).

Another nice thing about if you were to make a dedicated process for connecting and it was either caching or not (and could be swapped at runtime) is that mocking it would then become dead-simple.

Just remember, if in doubt about a design, you probably need to break it up into more actors/processes, they are fast and they scale.

Absolutely not, ETS is fast, designed to be fast.

Thanks for the reply

So, the library’s config.exs is loaded only if the developers of host applications import it correctly. I imagined this was possible, but it doesn’t seem very reliable. Still, good to know!

Ok, thanks. Good to know

I’m not sure what do you mean with this.

At the moment the cache is an ETS table created with [:set, :protected, :named_table, {:read_concurrency, true}]. It’s wrapped in a supervised GenServer started with the application.

Writes to the cache are go through casts to the owner process, so that they’re serialized (they’re going to be infrequent or, at best, in short bursts).
Reads, on the other hand, are executed in the invoking process, to minimize the data transferred between processes. I’m vaguely aware that an ETS table is actually a process itself, but I’m trying to support high levels of read concurrency and I’ve read that this approach is better than reading through GenServer.call(table_owner_pid, {:please_read, :my_key}).

Is this what you’re referring to? Or are you suggesting another process that either proxies to the cache or goes directly to Redis? And then I would swap this process if needed?

Ok, that’s interesting. I can think how I would do it by setting MIX_ENV in the terminal. Can you give me an example of how you’d do this with alias please?