How to use dynamic tags for excluding or including tests

Questions / Help
#1

Hi all,

I am working on my property based testing library propcheck. My latest feature is to re-run all failing properties (read: special test cases with their own macro definition) from the last test. So I write the failing property names to a file on disk and control the testing procedure in the next run depending on the file’s content. This works for now, but the reporting via ExUnit does not consider the skipped successful properties.

To achieve the regular ExUnit reporting, I generated a tag for each property in my macro with @tag failing_prop: true/false. The value of the tag depends on the last test run of this particular property. Log output shows that all tags of test module are evaluated before the tests are run, so this part works. But I never succeeded to filter out all properties which were successful at the last run (or vice versa to include only those which failed last time).

My questions are:

  • Is it generally possible to generate tags within a macro which sets value of the tag at runtime (= compile team of the .exs file)?
  • Is there any other possibility to inform ExUnit about skipped test cases (e.g. via an exception or a return value?
  • Is my macro construction regarding the tag simply wrong?

Here is the relevant snippet of the property macro:

quote bind_quoted: [name: name, block: block, var: var, opts: opts] do
      ExUnit.plural_rule("property", "properties")
      prop_name = ExUnit.Case.register_test(__ENV__, :property, name, [])
      %{module: module} = __ENV__
      if tag_property({module, prop_name, []}) do
        @tag failing_prop: true
      end
      def unquote(prop_name)(unquote(var)) do
        p = unquote(block)
        mfa = {unquote(module), unquote(prop_name), []}
        execute_property(p, mfa, unquote(opts))
        :ok
      end
  end

Thanks,
Klaus.