I just added some tests with external: true tags; not quite sure what to do with CI/CD; probably ‘warn but don’t fail’ for those tests.
kenny-evitt
0
kenny-evitt
kenny-evitt
I have a nice little test helper to seed the standard random functions from the Mix test seed so that random tests can be (exactly) reproduced, without needing to use a single fixed random seed.
0
arcanemachine
I’m still a noob when it comes to anything beyond basic testing. I haven’t figured out how best to handle tests to “external” APIs. I can mock them which is fine, but I like to have real tests that check against the real, actual service. That’s how I have envisioned the idea behind the external tag. (I’ve seen it used elsewhere too.)
1
sbuttgereit
Not sure that supports my “Integration Test” use case. The non-random, sequential test execution is a feature, not a bug, for this case; this is why not only is the random seed fixed, it’s fixed to “0”: this means the tests run in the specific order in which they are defined in the test script.
My “Integration Tests” prove the supported business processes, not the functions. Let’s say I have two functions create_user and authenticate_user. I do want to test that I can create the user, but, more importantly in this context, that the user created by create_user can be understood and works when processing function authenticate_user: the data must flow from one function to the next for the integration test to serve its purpose. It would be nice for the data values processed by these functions to originate randomly at the start of the integration test, but I want to see the results of earlier, data producing functions working well with later, data consuming functions.
My “Unit Tests” prove the functions, not the business processes. These tests are run asynchronously using a completely random seed and are focused on individual function workings. In this case, my example authenticate_user function can be tested in isolation at any time since I can establish pre-conditions (including pre-requisite data) for that specific test. It’s nice for any data seeding here to be random, too.
So in all cases, I’d like to have property-based-testing-like source data, but the testing goals have different needs when it comes to ordering execution.
1
sbuttgereit
Just off the cuff, sounds like a fine use of tags.
What you describe is a real testing scenario and it’s the kind of thing you often want on demand, not all the time.
One thing I do is specifically exclude tags so that I have to specifically request the run. I do that in test_helper.exs with something along the lines of:
test_kind =
if ExUnit.configuration() |> Keyword.get(:include) |> Enum.member?(:integration) do
ExUnit.configure(seed: 0)
:integration_testing
else
ExUnit.configure(exclude: [:integration])
:unit_testing
end
That excludes my integration tests so that if I don’t specify a test category that I get the unit tests only. Something similar would be useful to exclude external tests, too: you have to explicitly ask for testing external resources.
3
kenny-evitt
That’s exactly why I started using an :external tag in my own tests.
‘End-to-end’ tests are nice, but they’re also flakey, so it’s nice to be able to run them separately and NOT have CI/CD pipelines fail because the test service is down.
0
kenny-evitt
I wouldn’t write a test that depends on another one running (and passing) first/before. I’d just repeat the ‘earlier’ test(s) (or the relevant code) in the ‘dependent’ test.
(Really, I’d extract the common ‘setup’ code into a function that both tests can then call – mainly so I know that both tests should be ‘relying’ on the exact same ‘code behavior’.)
0