Btw, if you have some time (and you are interested), I would love to read a write up on those features from CommonTest that you are missing in ExUnit and vice-versa. If you are not comfortable sharing it publicly, feel free to ping me privately too.

I will try to find some time, but first I need to prepare my blog

For anyone who hasn’t used them, before @hauleth posts another great blog, a quick summary:

• EUnit: A unit testing framework in, for, and included with Erlang. In short it is generally designed to have test functions ‘inside’ a module, so in elixir-speak you could have something like this:

  defmodule MyModule do
    def add(a, b), do: a + b
  
    def add_test() do
      0 = add(0, 0)
      3 = add(1, 2)
      0 = add(-1, 1)
      -3 = add(-1, -2)
      -1 = add(1, -2)
    end
  end
  

  In essence EUnit takes any function whose name ends in _test with 0 arity and will run it. You could run it via :eunit.test(MyModule) and you will see it passes since all matches pass (having one of the match’s fail will return an error, which includes the stacktrace, output, and of course the final count of passed/failed tests). If a *_test/0 function returns successfully then the test is successful, if it doesn’t then that failure is reported.

  However, you don’t need to have the *_test/0 functions in the same module, instead you can move them to another, specifically into a module of the same module name with _test on the end of it, in the test/ directory, however that means that you can’t then test private code in the module, so EUnit sets a few defines to allow you to conditionally export_all code from a module so you can still test your private functions within tests.

  This other test/ directory can also have helper modules, data directories, and more. Just relying on if a function succeeds or not is not really fully useful either, so EUnit has a set of assert macro’s that you can use like ?assert(Expression) or ?assertMatch(Pattern, Expression) or ?assertException(Class, Pattern, Expression) and a lot of others too. Also fixtures, conditional specifications (whether a test should be run or not based on ‘something’), etc… etc…

  As you can see, Elixir’s ExUnit is very similar to EUnit, it’s basically the Elixir version of it and ExUnit could have quite easily been built on top of EUnit (though printing things in Elixir terms is quite useful).

  Now, the big thing about EUnit is the concept of Test Generators, these are functions ending in _test_ instead, and instead of using things like ?assert(...) you instead of ?_assert(...). In essence it makes the tests return functions that can be manipulated to generate tests on the fly, conditionally, and more (not in a macro-like way in Elixir but Elixir’s macro’s combined with ExUnit can make up this functionality pretty decently).

• Common Test: Now CT is not designed for unit tests, testing individual functions in isolation like EUnit is. CT is complex, it is designed for testing complex things, ExUnit obviously borrowed some from it as well but definitely diverges in a lot of ways.

  To start off with, CT tests are layered in a way similar to ExUnit, the CT test root is generally the tests/ directory, directories inside the tests/ directory are Test Object Directories (think of these like a ‘Context’ in phoenix terms, they test related functionality), inside those directories are modules called suite’s, they are modules named with _SUITE on the end, and inside those are case tests, each of which is a function, each suite module itself can have a corresponding data directory generally named <modulename>_SUITE_data.

  Now, a big BIG difference from CT to EUnit/ExUnit is that it creates little sandboxed areas to run tests in, so each test run gets its own unique/new priv directory, some state information passed in, etc…

  A suite should always have an all/0 function that returns a list of atoms corresponding to function names to call, those are the case’s of this suite, they each should take a single argument being the configuration (global data, private scratch directory, system setup/fixtures, etc…).

  You can trivially embed eunit inside CT for note, you just make a test case that calls :eunit.test(ModuleName), and this is what is often done.

  Now, when you run a CT test like via :ct.run_test([{:suite, MyModule_SUITE}]) it does a lot more than EUnit or ExUnit, it of course runs the tests, reports results, etc… etc… but it also generates directories named ct_run.MyNodeName.<datetime>/ that hold the private directories, run outputs, etc… of the tests, full of html files, CSS, etc… (with a top level index.html above these directories to easily access each within). These HTML results give a lot more information, what tests were run, how long they took, results, logs, etc…

  Back to where CT is similar to ExUnit, it’s setup functionality (fixtures essentially), unlike EUnit which are per-test or even per individual call, are more like ExUnit’s where they are essentially per-module or groups of tests, this works as you expect. CT’s setup/teardown functions can know which test functions are being run so you can customize setup based on those still (like having different setup functions for different groups of tests in ExUnit, just instead of being centralized like CT has it then ExUnit has the setup’s be distinct in different areas for each group). CT has grouping as well, similar to how ExUnit does it (unique setups and all) so you can choose whichever style you want in CT.

  However, in CT groups are significantly more powerful and configureable than in ExUnit, allowing for things like specifying whether they have to run one at a time, concurrently, what kind of ordering should be done, pure shuffled, etc… etc… along with a tree of configurations for the tests to define how they run (or not) together, whether to repeat tests an N number of times (until successful or if one fails or just in full), and still more!

  So yes, ExUnit has some aspects of CT and EUnit, but it does not have all the features of both, and ExUnit is conflating system testing with unit testing, which CT and EUnit keep distinct (and you can call EUnit from CT).

  In general in Erlang projects (this has been a blast from the past going though all this again!) you have EUnit’s, either as standalone *_test modules or in-module (I tended to do in-module, conditionally compiled, think doctests in ExUnit but in the Erlang world, and not docs) for testing functions and inputs/outputs and such, then you’d have all your CT suites in tests/ for doing integration testing, system testing, filesystem/network I/O testing, etc… etc…, with a suite generally dedicated just to running all your EUnit tests as well. Then you’d get a nice set of HTML files detailing everything that happened with history about each big test run, etc… You’d have system tests testing things in isolation, system tests specifically to test different systems in parallel to make sure no concurrency issues (generally repeated a number of times each time unless/until a failure happens), etc…

  In general there was a lot more than ExUnit. And yes although you can do system testing in EUnit, it generally gets hairy to deal with, have to setup all the concurrency tests yourself, handle your own looping, and sure you can do it in ExUnit too, but it is lacking a lot of the functionality to do so, so you’d have to hand-build it yourself, and you are still lacking the history and reports and sandboxed data directories and a lot more.

  And there is still more to CT than what I list here. It is expansive and I really wish Elixir built on both EUnit and CT. ^.^;

So, a quick summary, and yes that really was quick because both EUnit and CT are way powerful and capable. ^.^;

@OvermindDL1 @josevalim I have finally published my blog post:

@hauleth Are you still using Commoner? I was considering using mix for our Erlang project (also to prepare implementing some parts in Elixir), and running Common Test test suites is the one blocker I ran into so far. Are there other possible solutions?