marciok
Gust - A task orchestration system built in Elixir
Hi there! We created Gust: A task orchestrator inspired by Airflow.
For those who have never heard about Aiflow, it’s a Python-based workflow created many years ago by Airbnb.
After many years of using it, we realized that many of our frustrations, such as difficult setup, deployment, high resource consumption, and buggy UI, were symptoms of the system’s lack of high-performance parallel computing primitives. We concluded that we needed the right tool for the job: Elixir. The result was an orchestrator that is 10x more efficient and performant.
How is it positioned in relation to other background processing-related libs and projects?
They all intersect in some way, but each one focuses on different use cases.
Oban (OSS): It’s a background job library; compared to Gust, you would need to manually coordinate job dependencies, with no UI to visualize the task graph, logs, or code. It’s focused on individual jobs, not workflows.
Handoff: It’s a distributed DAG execution engine, no UI or persistence.
Reactor: It’s more similar to Gust, but its steps are not persisted (state is not saved in case of restart). Also, no UI, cron scheduling, and built-in triggers. It’s focused on computational graphs rather than full workflow orchestration.
OpenFn/lightning: Like Gust, it also possesses a UI and an engine to execute workflows; however, it’s not a DAG builder but an orchestrator between integrations, with no native DSL.
Gust combines a bit of each: a UI, task coordination, cron/other triggers, and built-in secret storage for use inside graphs. It’s a full-fledged orchestration system.
Suggestions and contributions are very welcome!
Trending in Announcing
Other Trending Topics
Latest Phoenix Threads
Latest Oban Threads
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 10 of 36 Posts
ghannam80
It look very interesting , Will it support a human in the middle tasks or steps ?
pjode
Looks cool! I noticed the
setup_gust_app.shand looking through it makes it feel like this is something you’d want to deploy independently separate. Is that fair to say. One thing I like about Oban, and Elixir in general, is that I run everything together with the option of separating it out later but within the same cluster. Is that possible?tfwright
Looks very powerful, but I am curious about the use case as someone maintaining Elixir apps. Oban is the de facto standard for “task/job management” and I already have a bunch of monitoring based around that, including a UI. It would be a tough sell for me to introduce an entire new service if it’s something I could make work with Oban. But maybe I’m not the target user, since none of the data flows I am currently dealing with are all that complex.
marciok
That’s one of our goals, something like a Camunda, but way simpler. It could start by having an “external upstream” task dependency, where it will halt the run until it is completed.
marciok
Yes! Gust in an umbrella app, that contains gust and gust_web apps which are different hex packages.
In practice, you could add gust into an existing app. The
setup_gust_appscript is just to facilitate the creation and configuration of an app that contains gust_web.The docs could be clearer on how to add gust or gust_web into an existing app.
nathanl
It’s good to have alternatives! But FYI, Oban Pro does have workflows: Oban.Pro.Workflow — Oban Pro v1.6.9
marciok
I get your point! Gust’s initial goal is to be an Airflow alternative, not a background job lib.
At its core, Gust is a DAG orchestrator; it really shines when you have complex workflow logic that processes unstructured data, where you need quick iteration.
Example: “You’re ingesting unstructured data from dozens of websites (HTML pages, PDFs, APIs with inconsistent schemas) and turning it into structured data.”
Overall, the Gust project is an umbrella app, with the UI and processing logic split; in theory, it is possible to use the Gust UI to monitor Oban jobs or create workflows with them.
code-shoily
Thank you so much for building and sharing this. Looks fantastic!
marciok
Indeed! But it comes at $150/mo;
sorentwo
It does. It’s paid because we provide a service that people rely on. It is actively developed and supported. That’s our job.
There’s no chance of Oban (or Pro) going unmaintained, unsupported, or abandoned because we burned out or got a new job that doesn’t offer 20% time.
P.S. It’s $135/mo with the annual plan