Gust - A task orchestration system built in Elixir

Your Libraries & OS Mentoring Libraries
, , , ,
1

Hi there! We created Gust: A task orchestrator inspired by Airflow.

For those who have never heard about Aiflow, it’s a Python-based workflow created many years ago by Airbnb.

After many years of using it, we realized that many of our frustrations, such as difficult setup, deployment, high resource consumption, and buggy UI, were symptoms of the system’s lack of high-performance parallel computing primitives. We concluded that we needed the right tool for the job: Elixir. The result was an orchestrator that is 10x more efficient and performant.

How is it positioned in relation to other background processing-related libs and projects?

They all intersect in some way, but each one focuses on different use cases.

Oban (OSS): It’s a background job library; compared to Gust, you would need to manually coordinate job dependencies, with no UI to visualize the task graph, logs, or code. It’s focused on individual jobs, not workflows.

Handoff: It’s a distributed DAG execution engine, no UI or persistence.

Reactor: It’s more similar to Gust, but its steps are not persisted (state is not saved in case of restart). Also, no UI, cron scheduling, and built-in triggers. It’s focused on computational graphs rather than full workflow orchestration.

OpenFn/lightning: Like Gust, it also possesses a UI and an engine to execute workflows; however, it’s not a DAG builder but an orchestrator between integrations, with no native DSL.

Gust combines a bit of each: a UI, task coordination, cron/other triggers, and built-in secret storage for use inside graphs. It’s a full-fledged orchestration system.

Suggestions and contributions are very welcome!

35 Likes
2

It look very interesting , Will it support a human in the middle tasks or steps ?

2 Likes
3

Looks cool! I noticed the setup_gust_app.sh and looking through it makes it feel like this is something you’d want to deploy independently separate. Is that fair to say. One thing I like about Oban, and Elixir in general, is that I run everything together with the option of separating it out later but within the same cluster. Is that possible?

1 Like
4

Looks very powerful, but I am curious about the use case as someone maintaining Elixir apps. Oban is the de facto standard for “task/job management” and I already have a bunch of monitoring based around that, including a UI. It would be a tough sell for me to introduce an entire new service if it’s something I could make work with Oban. But maybe I’m not the target user, since none of the data flows I am currently dealing with are all that complex.

1 Like
5

That’s one of our goals, something like a Camunda, but way simpler. It could start by having an “external upstream” task dependency, where it will halt the run until it is completed.

2 Likes
6

Yes! Gust in an umbrella app, that contains gust and gust_web apps which are different hex packages.

In practice, you could add gust into an existing app. The setup_gust_app script is just to facilitate the creation and configuration of an app that contains gust_web.

The docs could be clearer on how to add gust or gust_web into an existing app.

7

It’s good to have alternatives! But FYI, Oban Pro does have workflows: Oban.Pro.Workflow — Oban Pro v1.6.9

1 Like
8

I get your point! Gust’s initial goal is to be an Airflow alternative, not a background job lib.

At its core, Gust is a DAG orchestrator; it really shines when you have complex workflow logic that processes unstructured data, where you need quick iteration.

Example: “You’re ingesting unstructured data from dozens of websites (HTML pages, PDFs, APIs with inconsistent schemas) and turning it into structured data.”

Overall, the Gust project is an umbrella app, with the UI and processing logic split; in theory, it is possible to use the Gust UI to monitor Oban jobs or create workflows with them.

4 Likes
9

Thank you so much for building and sharing this. Looks fantastic!

10

Indeed! But it comes at $150/mo; :grinning_face_with_smiling_eyes:

2 Likes
11

It does. It’s paid because we provide a service that people rely on. It is actively developed and supported. That’s our job.

There’s no chance of Oban (or Pro) going unmaintained, unsupported, or abandoned because we burned out or got a new job that doesn’t offer 20% time.

P.S. It’s $135/mo with the annual plan :wink:

15 Likes
12

It would be great if such use case or similar presented as sample use case project

1 Like
13

Oban is awesome! Gust has a different focus; the two can absolutely coexist & complement each other.

6 Likes
14

Yes, I’ll include examples and use cases.

1 Like
15

Gust is awesome. Well done. I’m working on a similar project, but it’s more inspired by Dagster:

Grateful that you’ve shared Gust, as it’s given some inspiration on how I can improve my project.

2 Likes
16

Nice :flexed_biceps:t2:! It looks promising. I’ve added a star on your repo, looking forward to seeing it evolve.

1 Like
17

How does this library compare to Handoff? Handoff v0.2.0 — Documentation

2 Likes
18

Gust is a full workflow orchestration engine with UI, persistence, and scheduling built in. Handoff, on the other hand, is more of a lib for DAG execution. They are similar in concept, but with a different focus.

BTW, I sent you a DM on X @polvalente a few days ago to explore how both tools might complement each other. :slight_smile:

2 Likes
19

Awesome! Messaged you back :slight_smile:

Thanks for the comparison

1 Like
20

Hi @marciok
I am a Data Engineer and right now, I am testing the Dagster as an alternative to Airflow. And while I was doing so, I thought to myself: It would be nice to see something related to data orchestration with Elixir tooling. And now I stumbled upon your lib.

I´ll take a look asap.