I don’t use Swarm, but your problem seems like a more general case where secrets must be provided when starting the app (as opposed to building the release), and they might need to be derived from various sources, not necessarily OS env. In my cases, I used etcd and custom json files to fetch stuff, so various OS env impros wouldn’t work for me anyway.

The way I deal with this (which doesn’t work for everything, but does for most things, including Phoenix and Ecto), is:

1. In application start callback, prior to starting the top-level supervisor, I fetch secrets from wherever.
2. I merge secrets into proper places in app env (e.g. repo or endpoint config).
3. I start the supervision tree.

Hi, I’m one of the maintainers of KafkaEx. There’s a lot of good feedback here. I agree that the way worker connections are supervised by default could be better - it’s a piece of the design I inherited with the project. If I understand correctly, that decision was originally made to make it “easy” to start an application and connect to Kafka. I use KafkaEx in production and have had to deal with these complications myself as well.

Unfortunately, we are now in a situation where there are lots of ideas how to make the API better but there are people already using it, so we need to be careful not to break it for them. I’ve considered pushing for an API re-design with a major version change (we are still < 1.0), but we also have several key Kafka features that still need to be implemented that are more urgently needed. We welcome any help we can get!

We do provide the flexibility to disable the default worker and to start workers under your own supervision tree. As was mentioned, however, this is a bit cumbersome and confusion-prone. Anyone who has trouble with KafkaEx is welcome to hop on to the Elixir-lang slack and ask in the #kafkaex channel - we try very hard to be available.

It should just not start then, it should supply a supervisor that you would add to your own supervision tree with whatever options you want to give it.

It should not be an ‘application’ type library unless it is truly standalone, otherwise it should be a supervised type library.