This is great, and something I think is very much needed in the ecosystem!

I believe that this has potential to be used beyond just starting containers from test. For example, we could use this to start/link the accompanying containers when the system starts, or to run containerized one-off commands.

I’ve been meaning to develop something like this myself as the part of the ci library, but it looks like excontainers already solves many of the things I planned.

The usage of ryuk for container cleanup is nice, I didn’t even know about it! My plan was to start a separate OS process (ideally a hidden Erlang node), which would roughly do the same thing. I still plan on trying this out, because I think it could solve a subtle race condition that exists with ryuk.

I wonder why is Docker.Api marked with @moduledoc false? This is something that I think could be useful to the general population, and I think it’s possibly the most important part of this library. Having a simple docker API wrapper would allow people to build various custom abstractions & docker flows.

The synchronous logic based on polling in exec_and_wait likely won’t work for the scenarios I have in mind. I’d probably aim for a reactive approach, i.e. a GenServer that receives response chunks from the start request, and stops once the exec command has finished. That way the client could also handle stdout/stderr in realtime, e.g. by logging the output. I’m not sure if this can be done with tesla, but I think that it should be possible with mint. I’ll try this out myself, because I need this for the ci library, but I can’t commit on time.

In any case, great job, thank for making this!

It’s a general race condition, but very subtle. Consider the following sketch:

container_id = start_my_container()
Excontainers.ResourcesReaper.register({"id", container_id})

If the beam node is taken down after docker run is invoked, but before the container is registered, the container will not be reaped.

Admittedly such scenario is not very likely, but given time it’s bound to happen at some point, and it may in turn lead to bugs which are hard to track.

The only solution I can think of is to run an external program which would be both, the starter of docker containers, and the “garbage collector”. When we want to start the container we ask the program to do this for us (which would obviously be wrapped under some nice API). That way, even if the beam node is taken down in the middle of the container start, the external program can still clean up all the resources. My idea is to try implementing this in Elixir, rather than go. If this works, we could discuss the option of merging it to excontainers.

No need to change your priorities for the moment My plan is to implement my own docker exec as a wrapper around OsCmd. This would give me all the features of OsCmd practically for free, most notably async & sync execution with reactive output capture and proper cleanup on early shutdown. As a consequence, such implementation would use CLI instead of the HTTP API, so I don’t think it makes sense to have it in excontainers.

You are right about the race condition, thanks for letting me know!

I’m not sure if this would fit your scenario, but perhaps a simpler solution could be to use something different than the container ID to register the container for reaping, so that we can set up the reaper before starting the container (Ryuk supports all kinds of filters – by name, label, etc.)
For example we could generate a unique identifier (e.g. a UUID) for the container, set the reaper to kill containers with label e.g. reaping_id=<my-id> and then start the container with the unique label.

My idea is to try implementing this in Elixir, rather than go. If this works, we could discuss the option of merging it to excontainers.

Sure! I’d be happy to make excontainers 100% elixir, even for containerized dependencies

My plan is to implement my own docker exec as a wrapper around OsCmd […]

Yeah I agree both that excontainers would not be a good fit for your usecase, and that bringing CLI-based functionalities in excontainers would not make much sense.

I’ll let you know if I make substantial progress on that front thanks again for your feedback!