Hi there! We created Gust: A task orchestrator inspired by Airflow.

For those who have never heard about Aiflow, it’s a Python-based workflow created many years ago by Airbnb.

After many years of using it, we realized that many of our frustrations, such as difficult setup, deployment, high resource consumption, and buggy UI, were symptoms of the system’s lack of high-performance parallel computing primitives. We concluded that we needed the right tool for the job: Elixir. The result was an orchestrator that is 10x more efficient and performant.

How is it positioned in relation to other background processing-related libs and projects?

They all intersect in some way, but each one focuses on different use cases.

Oban (OSS): It’s a background job library; compared to Gust, you would need to manually coordinate job dependencies, with no UI to visualize the task graph, logs, or code. It’s focused on individual jobs, not workflows.

Handoff: It’s a distributed DAG execution engine, no UI or persistence.

Reactor: It’s more similar to Gust, but its steps are not persisted (state is not saved in case of restart). Also, no UI, cron scheduling, and built-in triggers. It’s focused on computational graphs rather than full workflow orchestration.

OpenFn/lightning: Like Gust, it also possesses a UI and an engine to execute workflows; however, it’s not a DAG builder but an orchestrator between integrations, with no native DSL.

Gust combines a bit of each: a UI, task coordination, cron/other triggers, and built-in secret storage for use inside graphs. It’s a full-fledged orchestration system.

Suggestions and contributions are very welcome!

https://github.com/marciok/gust