How to enforce a suite of tests to be implemented

Questions / Help
,
1

I am running a team of developers that implement some payment gateways. I want to be able to enforce some tests scenarios that need to be implemented. I came up with this solution but I am not sure it’s too nice or the best one.

I wrote this macro:

defmodule IntegrationTests.PaymentGatewayTests do
  use ExUnit.CaseTemplate

  using do
    quote location: :keep do
      @behaviour IntegrationTests.PaymentGatewayBehaviour

      import Phoenix.ConnTest
      import Plug.Conn

      setup do
        setup_test()
      end

      @tag timeout: 2 * 60 * 1_000
      test "test1", context do
        test1_implementation(context)
      end
    end
  end
end

the behaviour:

defmodule IntegrationTests.PaymentGatewayBehaviour do
  @callback setup_test :: {:ok, list}
  @callback test1_implementation(context :: map) :: any
end

and this is how it should be used:

defmodule IntegrationTests.Payments.AdyenTenderTest do
  use IntegrationTests.PaymentGatewayTests
  
  @impl true
  def setup_test do
    # setup all the test data
    {:ok, conn: put_req_header(build_conn(), "accept", "application/json")}
  end

  @impl true
  def test1_implementation(context) do
    #implement the test
  end
end

Some things I do not like: you can not run an individual test, some training is needed to explain others how to use it, setup is common for all tests and you may have different setup data.

I would like some input, or some suggestions on how to enforce those tests.

2

The setup part I solved by using a named macro, which handles setup only for common tests instead of having use Module already include everything.

The macros:

Usage:

1 Like
3

You make a pretty convincing argument for not doing this, TBH :slight_smile:

It’s obviously hard to tell from the shortened example in your post, but what does IntegrationTests.PaymentGatewayTests do? As shown there, it’s delegating all the setup and test implementation to the implementing module - mostly indirection. That’s different from the code @LostKobrakai linked to, which uses the macro to ensure that every adapter gets exactly the SAME tests with custom per-adapter setup.

1 Like
4

What I am trying to achieve is to force developers to implement those tests. My test cases from the PaymentGatewayTests calls methods that need to be implemented inside the test file. If they are not the compilation will fail.
I am open to suggestions since I am not sure how I can do that. What I really want is to have some kind of behaviour will all the test cased that need to be implemented.

5

What if your developers do use ExUnit.Case instead of use IntegrationTests.PaymentGatewayTests? I’m not sure there’s much safety in that especially if you seem to not trust your people doing their jobs well.

6

you may be interested in this as a starting point:

It doesn’t peek any further to introspect that the test modules actually do what they are supposed to do, but it will at least fail CI if someone hasn’t at least started the skeleton of the test. You could probably do more, by checking the list of functions in those modules to see that it’s implemented a test that you would expect, and if you wanted to do even even more, you could probably look at coverage reports to confirm that the corresponding modules have coverage.

BTW, if you want to get confirmation on function names, you can use the <module>.__info__(:functions) to obtain f/a tuples, you could do something like Atom.to_string/1 followed by either =~ or make the format of the test name explicit and required.

7

It’s much easier for me on code review to check that they added the proper use statement instead of reading all the tests to see if they covered all the test cases.

8

Not duplicating all the test cases makes sense. But using a standalone macro vs. the ex_unit case template should be almost the same in terms of reviewability, while the first one gives you more flexibility even if just for ordering the common tests after implementation specific setup calls.