Are mix tests integration tests by default?

stefannovak · August 10, 2024, 12:27pm

Hi all, noob question sorry. I’m coming from a C# .NET background and I’m interested in the different ways things are done in these two ecosystems.

I’ve begun testing my first Elixir application. What’s surprised me is ‘mix test’ starts up my entire application. Does this mean that by default you’re doing full integration tests with ‘mix test’?

I had a very simple module that I wanted to unit test, so I was running that with ‘mix test --no-start’. I feel like that’s the flag you use to unit test, and ‘mix test’ is for the integration tests.

So today I’m figuring out setting up my test environment, creating a test postgres user and Db, and creating a test user for Rabbit. What if I had more dependencies, some third party API that is costly to use, like some AI Form Recogniser. If I just wanted to test some validation on a message from Rabbit being consumed by my application which is right at the start of my application, the test won’t stop until that message is processed by the entire rest of the application right?

If anyone wanted to pull my code, they’d have to set up every single dependency, both dev and test, to run and test my code?

I’d really appreciate some insight into this Coming into this functional paradigm I found testing really appealing, as everything is functions and functions are generally easier to test than some bloated class in C#.

D4no0 · August 10, 2024, 1:02pm

Yes, it starts the application, but that doesn’t mean that you cannot set things you don’t want to be started when running tests. You can use configuration to set what functionality you want to be active on test vs other environments.

The functions can have runtime properties, for example:

def call_global_genserver() do
  GenServer.call(:my_global_genserver, :hello)
end

In this case this global genserver will be most probably be started by your application supervisor, so you want to start the application when running tests, otherwise you will have to somehow do this setup manually. This is not an integration test, but at the same time this is not a pure function.

Nobody said you have to run external services to test your application, the same testing rules apply as in other languages, you can run the tests with real dependencies or you can mock them, depends on the use-case and what you are looking for in those tests.

There isn’t a silver lining that every project uses, but the rule of thumb is that usually for example you don’t mock the database calls as you lose a lot of insights related to potential errors.

Once again, depends on configuration, you can easily configure to start different things in different environments, check elixir config documentation, there you have configs for specific environments. In what way you decide to organize those will dictate how easy it is to work on the project and how good your tests will be.

LostKobrakai · August 10, 2024, 1:25pm

While there‘s nothing enforcing you to test things in a unit level style, there‘s also nothing preventing you from writing such tests. For a functional core of an application I‘d suggest doing that.

But there‘s benefit in testing things at different levels, where interacting with processes started by the application may be useful. As always in technology it‘s tradeoffs and sometimes it‘s not worth to start individual instances of processes per test with the additional complexity for sometimes little gain.

pdgonzalez872 · August 10, 2024, 4:07pm

Welcome @stefannovak !

You said you were new (if you know this already, hopefully other folks that don’t can benefit from this), so I’ll link you to some interesting things that help a ton when running tests from an operational side.

As you said, not all tests are equal. Sometimes you have third party stuff that you need creds for, sometimes you have great tests that test a bunch of things together but it’s slow (and if THAT test fails, things are bad ) In these cases, it’s good to leverage the mix test options (which I find absolutely excellent) coupled with tags (called filters) in the test files.

Here is a link to the options: mix test — Mix v1.17.2.

In order to use those well, you should know about tags and filters and you can use them both modules and also tests.

Here is where tags are talked about: ExUnit.Case — ExUnit v1.18.0-dev

Here is a link to some filter usage: mix test — Mix v1.17.2

In a short example, you could do something like this (excerpt from ExUnit.Case — ExUnit v1.18.0-dev):

defmodule ApiTest do
  use ExUnit.Case
  @moduletag :external

  # rest of module with tests
end

And then, you can run/skip the above with the following:

mix test include --external
mix test exclude --external
mix test only --external

Maybe you have a use case like this where you may not want to run all slow tests by default for some reason, you want fast feedback and you have known slow tests (check out --slowest to find them )

defmodule MyApp.SomeTest do
  use ExUnit.Case

  @tag :slow
  test "everything works as expected" do
    # set up a BUNCH of data to make sure different domains work as expected in unison
  end
end

You can configure defaults for tests to be run in test_helper.exs, like done here: avalanche/test/test_helper.exs at main · HGInsights/avalanche · GitHub, you can exclude the slow ones if you’d like.

Here is an example of usual steps to do when you have those integration tests: GitHub - HGInsights/avalanche: Avalanche is an Elixir Snowflake Connector built on top of the Snowflake SQL API v2..

Maybe you can add an example module to mix test — Mix v1.17.2? I don’t see an example of setting this up to discuss the filters, so I think that would be helpful in the docs. Doc improvements are always welcome and a great way to get going in the community. The maintainers are incredible. You mentioned you came from C#, so maybe that’s a different change in mindset as well since in the Elixir community (at least how I see it), if you can help, do it! :). Please reach out to me if you need help creating a PR for the docs.

At some point, you’ll get in a situation where you want to run tests but don’t want to set up the external api keys and such. Give this a read when that happens: Mocks and explicit contracts - Dashbit Blog and know that Mox is . Mostly, because of the concept of boundaries. Craft your boundaries well and your tests (and Mox) will just work TM

Hopefully this helps you a little! Welcome to the community!

stefannovak · August 10, 2024, 4:43pm

Thank you! This really helped clear things up for me, I had no idea how in depth ExUnit and Mix Test are, there’s so many configuration options that I’m not used to before. I just did ‘dotnet test’ and the rest took care of it all! Very cool! Thanks

stefannovak · August 10, 2024, 4:48pm

Hey thanks for the reply and the warm welcome! I had actually been looking around GitHub for an example Elixir project, something not too big or small, that I could take some inspiration from. That Avalanche repo looks like exactly that, so thank you for that!

I had no idea how in depth ExUnit is, to be honest I hadn’t read the docs of it, I instead was looking at the test docs for Phoenix, Mix, GenServer, etc. Tonight I’m going to eat some chinese food and just sit down and read this whole hex doc!

I was wondering about Mocks too, as there’d surely be a time I want to control what HTTP response I get, and of course the article you linked is written by José, he’s everywhere!

I do hope to contribute to Elixir, I’ve been finding it’s a fantastic ecosystem for developers, I love it. I think I’m a bit far away from contributing to Mix at the moment though Maybe that’ll be my goal for the end of the year though.

Thanks for all the helpful links!

D4no0 · August 10, 2024, 10:26pm

The best thing you can do now is just to take some time to understand how to do things with elixir and where its strengths lie. A lot of people come to the ecosystem and expect the things to work like they were used to in their previous languages, but if you open your mind to new things the ecosystem provides, it will help you become an overall better engineer.

Here is one more project you can look at: SSL MOON - SSL Monitoring and Observability of Open Networks . It’s very far from being done or at a level that I would like it to be, but it should be small enough for people to understand what is going on without going too much in detail on complex things, also any contributions will be more than welcome and I will gladly guide on how to do them.

dimitarvp · August 11, 2024, 12:40am

As you are discovering, Elixir by default does unit tests but also starts various processes that you might need – a prominent example is the database repo, but also Phoenix endpoints, GraphQL resolvers, and others.

The idea is to be as close to testing the real thing as possible, more or less.

Furthermore, the whole “unit vs. integration” tests thing is kind of artificial as I think many of us would agree; “integration” is more like “leaky tests, where more moving parts not under the control of our application are involved”. Whereas “unit” is viewed more as “pure or almost pure tests where we control everything”.

I never saw both terms as something different, they are kind of describing how much of your testing environment you have under control; kind of how you would call water “clean” and “dirty”… but both are still water.

And obviously both kinds of tests have their important place in the picture.

Nowadays I know nothing about C# (worked with it briefly no less than 10 years ago) but as @D4no0 said – please don’t expect things to work like in there. That practically never happens even between languages that are very similar, too.

Take the requisite time to understand what an OTP process is and then mix test actually starting your app would make much more sense.

sodapopcan · August 11, 2024, 2:35am

I’m also in the boat of people who are fuzzy on what’s integration vs unit at this point.
Like, any single function at any layer could be considered a “unit,” right? Of course I know what you mean, but still, I don’t think about it too much now.

sums it up for me.

You also didn’t mention what type of work you do. If you are a web developer interested in LiveView, though, testing LiveView is so great. I could be greaterer (like if someone can figure out how to allow JS interactions), but it is currently very great as is. If you’re into BDD then fuggetaboutit!—there’s nothing better than starting with a LiveView test.

I write three kinds of test for my web apps, eschewing the typical testing lingo (even if it may line up pretty well):

Context tests
LiveView tests (or Controller tests)
Wallaby tests mostly covering for JS interactions, or just the happy path of some “user journeys”.

Mox has been mentioned and there are several other mocking libs out there, but there is a good-lookin’ new kid on the block called Efx which I suggest you check out. Not only is it very simple, it doubles as a way to flag your effectful (as in side-effects) functions as such, even if you aren’t going to mock them, if that is something you care about (which you very well might the deeper you fall in love with FP).