Excontainers - Throwaway containers for your tests

[dallagi]

TL;DR: library to conveniently spawn throwaway docker containers for your integration/functional tests. Not stable yet. Written by an Elixir newbie, any feedback is super welcome!

Hello everyone

I’ve been studying Elixir in the past few months, and to get some practice I decided to try and develop a library that I felt could be useful for someone.

I developed something similar to the Testcontainers java library for elixir.
The idea is to make it as convenient as possible to launch and use docker containers within your integration/functional tests.
No need for external docker-compose.yml files, your dependencies are nicely and declaratively defined within your own tests.

Repo: https://github.com/dallagi/excontainers
Hex.pm: https://hex.pm/packages/excontainers

Installation, features and basic usage are documented in the project’s README.

Note: while the functionality is there and it is quite comprehensively tested, this library hasn’t seen any real world usage yet, so it is to be considered not stable yet.

I hope someone will find it useful! If you’d like any functionality that is not implemented yet you are welcome to open an issue.
As I mentioned I just started learning elixir, so if you have any feedback or suggestion on how to make the project better/more idiomatic, please let me know! Thanks a lot in advance

[sasajuric]

This is great, and something I think is very much needed in the ecosystem!

I believe that this has potential to be used beyond just starting containers from test. For example, we could use this to start/link the accompanying containers when the system starts, or to run containerized one-off commands.

I’ve been meaning to develop something like this myself as the part of the ci library, but it looks like excontainers already solves many of the things I planned.

The usage of ryuk for container cleanup is nice, I didn’t even know about it! My plan was to start a separate OS process (ideally a hidden Erlang node), which would roughly do the same thing. I still plan on trying this out, because I think it could solve a subtle race condition that exists with ryuk.

I wonder why is Docker.Api marked with @moduledoc false? This is something that I think could be useful to the general population, and I think it’s possibly the most important part of this library. Having a simple docker API wrapper would allow people to build various custom abstractions & docker flows.

The synchronous logic based on polling in exec_and_wait likely won’t work for the scenarios I have in mind. I’d probably aim for a reactive approach, i.e. a GenServer that receives response chunks from the start request, and stops once the exec command has finished. That way the client could also handle stdout/stderr in realtime, e.g. by logging the output. I’m not sure if this can be done with tesla, but I think that it should be possible with mint. I’ll try this out myself, because I need this for the ci library, but I can’t commit on time.

In any case, great job, thank for making this!

[dallagi]

Hi @sasajuric , thanks a lot for your feedback and for having a look at the code!

I believe that this has potential to be used beyond just starting containers from test

I completely agree with you.
I started with a focus on the testcontainers usecase in order to deliver something potentially useful as soon as possible and get feedback from the community.
However my goal is to do as you suggest and evolve the library to eventually make it useful for more generic scenarios as well.

I think it could solve a subtle race condition that exists with ryuk

Is that race condition specific to your use case, or is it something I should care about for Excontainers too? In the latter case could you tell me more please?

I wonder why is Docker.Api marked with @moduledoc false?

Same reason as above - so far I tried to keep the scope as narrow as possible. But I do plan to work on making the library more generic as soon as I have time.

The synchronous logic based on polling in exec_and_wait likely won’t work for the scenarios I have in mind.

Yeah polling is definitely not optimal, but I thought it was an acceptable shortcut in the initial phases of development.
I plan to try and replace it with a more efficient reactive approach, which would also be useful for other use cases (e.g. waiting for a specific log to detect when the container is ready).

And by the way your CI library is awesome, if this is a feature that could help with your effort I would be happy to make it my priority for the next steps of Excontainers

Thanks a lot again!

[sasajuric]

@dallagi

Is that race condition specific to your use case, or is it something I should care about for Excontainers too? In the latter case could you tell me more please?

It’s a general race condition, but very subtle. Consider the following sketch:

container_id = start_my_container()
Excontainers.ResourcesReaper.register({"id", container_id})

If the beam node is taken down after docker run is invoked, but before the container is registered, the container will not be reaped.

Admittedly such scenario is not very likely, but given time it’s bound to happen at some point, and it may in turn lead to bugs which are hard to track.

The only solution I can think of is to run an external program which would be both, the starter of docker containers, and the “garbage collector”. When we want to start the container we ask the program to do this for us (which would obviously be wrapped under some nice API). That way, even if the beam node is taken down in the middle of the container start, the external program can still clean up all the resources. My idea is to try implementing this in Elixir, rather than go. If this works, we could discuss the option of merging it to excontainers.

@dallagi

if this is a feature that could help with your effort I would be happy to make it my priority for the next steps of Excontainers

No need to change your priorities for the moment My plan is to implement my own docker exec as a wrapper around OsCmd. This would give me all the features of OsCmd practically for free, most notably async & sync execution with reactive output capture and proper cleanup on early shutdown. As a consequence, such implementation would use CLI instead of the HTTP API, so I don’t think it makes sense to have it in excontainers.

[dallagi]

You are right about the race condition, thanks for letting me know!

I’m not sure if this would fit your scenario, but perhaps a simpler solution could be to use something different than the container ID to register the container for reaping, so that we can set up the reaper before starting the container (Ryuk supports all kinds of filters – by name, label, etc.)
For example we could generate a unique identifier (e.g. a UUID) for the container, set the reaper to kill containers with label e.g. reaping_id=<my-id> and then start the container with the unique label.

My idea is to try implementing this in Elixir, rather than go. If this works, we could discuss the option of merging it to excontainers.

Sure! I’d be happy to make excontainers 100% elixir, even for containerized dependencies

My plan is to implement my own docker exec as a wrapper around OsCmd […]

Yeah I agree both that excontainers would not be a good fit for your usecase, and that bringing CLI-based functionalities in excontainers would not make much sense.

I’ll let you know if I make substantial progress on that front thanks again for your feedback!

[sasajuric]

@dallagi

a simpler solution could be to use something different than the container ID to register the container for reaping, so that we can set up the reaper before starting the container

Oh, I see now that in the code that filters are “lazy”. Yeah that’s a great idea that would definitely work! Maybe consider mentioning this approach in docs.

I’ll still give this a try, b/c I want explore some ideas, but it looks like using custom unique labels will work.

[NobbZ]

Okay, I am currently looking into this, though wondering about best strategies to migrate the throwaway database.

Also I’d really like a shared_container that I could start once and that would stay for all the tests and not those in a single module.

[dallagi]

Hi @NobbZ , thanks for your feedback.
Looking back at what you wrote here, I’m starting to think that excontainers may actually not be a perfect fit for your needs.

Excontainers is mostly useful to spawn short-lived containers to allow running integration tests asynchronously and with good test isolation.
Using it to spawn long-running containers is possible, but not very straightforward.

Of course any suggestion to make it more suitable for your scenario is very appreciated
Some potential problems I see are:

• we’d need a way to start containers and run migrations (if any) before the application starts, otherwise other processes that depend on these containers could start to break. This may not be a big deal in some cases, as everything could go back to normal once migrations are done, but even then it would still be ugly.
• if you need the container to listen on a specific port on the host (eg. postgres on 5432) then it would be hard to handle multiple instances of your application (eg. one for your local server and the other to run tests / run a REPL). New instances of your app would try to spawn the container and fail due to the port being already taken.

@NobbZ

wondering about best strategies to migrate the throwaway database

In case of a long-running container shared by the whole test suite, you could start the container and run migrations in your test_helper.exs.

If you have containers under your supervision tree then it could be a little more tricky, as you’d need some way to execute migrations once the process (and hence the container) is started.

@NobbZ

Also I’d really like a shared_container that I could start once and that would stay for all the tests and not those in a single module.

This is something that I’ll consider to add.
For now the easiest ways to start a permanent container are either to start it manually in your test_helper.exs or to place {Excontainer.Container, your_container_configuration} under the supervision tree of your app.