What do you use for CI/CD pipeline configuration? Dagger/Earthly/YAML?

I am not in the loop but I just took a look at Earthly’s pricing page and the free tier is still there. What am I missing? Is it going to be mandatory to pay for GitHub Actions?

This is from their Slack channel. As long as you’re not using cloud satellites I’m guessing you should be fine, as long as you don’t expect any further development.

Hey, Earthly community - some of you have been asking for updates, and while we still don’t have all the specifics, I wanted to shed some light on what’s been happening behind the scenes.

We’ve been working on ways to build a sustainable venture-scale business. Our initial plan was to monetize Earthly Satellites, which speeds up builds. However, we faced two major challenges: adoption of open-source technologies is typically slow (think Docker’s decade-long journey), and we found ourselves competing with our own free, self-hosted version of remote BuildKit. We can’t afford to follow the path of other open-source companies like Hashicorp, where monetization doesn’t happen until Series C - especially in today’s economic climate with budget cuts and funding challenges.

As we explored alternatives, we realized that the adoption of Earthly is often hindered by the perceived effort of migrating existing pipelines. For companies with hundreds of engineers, moving to Earthfiles can feel overwhelming, especially when teams have other pressing priorities. Even our most ambitious ideas, like serverless distributed builds for faster CI/CD (a.k.a. compute v2 internally), wouldn’t solve this core issue.

We then developed ucacher, a drop-in technology that can automatically cache existing CI scripts without syntax changes. While promising, it presented some technical challenges in certain edge cases, and we found that CI speed isn’t a top concern for most enterprises. This led us to a new idea: Earthly Governance, aimed at managing CI/CD in large organizations facing challenges with tech stack diversity. Early feedback from companies with over 200 engineers has been very encouraging, and with some new unannounced capital that we raised recently, it can help us reach monetization at scale.

While we’re excited about this new direction, it does mean changes for Earthly open-source and Satellites. We will sustain open-source Earthly with bug fixes but don’t have concrete plans for new feature development. We may also shut down Cloud Satellites sometime in the next 12 months. If we go down this path, we’ll provide ample notice to support your transition, and we will also offer free alternatives so that your builds are not interrupted. We know this isn’t the news you were hoping for, but we want to be transparent so you can plan accordingly. We’re here to assist in any way we can as we move forward.

I see, thanks for following up.

At least they said they’ll provide free alternatives when the time comes so I would not be too worried.

But still, no idea, sorry for the noise. I’d also be interested in an answer however. I’ve been extremely unhappy with the YAML nature of CI/CD for ages but never had the time and energy to seriously tackle the problem.

I personally hate yaml configuration files with passion too. These last months I configured entire CI + build pipelines for elixir projects both in CircleCI and github actions.

CircleCI is far the worst one I’ve met yet, starting from the fact that it takes usually at least 40 seconds to start the required docker containers for each pipeline, to the fact that the syntax they have is abysmal and poorly documented. This was the first time I had to resort to using chatGPT to actually generate me configs I can use to poke to understand how it works, after digging into the documentation for a few hours without any success.

Github actions is better in the regard of reusable pipelines and it’s obviously more streamlined, but their design imposes some hard limitations too. In terms of documentation, horrible just like CircleCI. Also, if you are using paid runners hosted by them, be careful as you will be scammed when it comes to time billed (sadly can’t find the link for more information on that).

Gitlab is the one I used from the start of my career and is actually the one I prefer the best. You can deploy and connect a custom runner in 5 minutes on a clean VM and it just works. You can also take a look at a generic configuration for elixir projects, much cleaner and shorter than both github and circleCI. If I remember correctly you also have validations for your configuration if you use the web editor from gitlab, which is pretty nice.

Overall, I would prefer for all these YAML configurations to be discarded in favor of a DSL or programming language that can validate both the structure as well as if those configurations make any sense from the logical standpoint.

I don’t care much about being able to run locally CI/CD pipelines, simply because in commercial projects you have a lot of secrets that are being setup in the repo, so for you to run them locally, you have to leak all those values.

I however would like to be able to run tests on configuration files, so as you mentioned, don’t have to push every-time to repo to check if the config is valid. I have not so long ago set a brand new CI pipeline in github and I have 30+ commits on that MR, really idiotic practice.

We can only pray somebody will finally wake up and do it. Us the working programmers don’t get much time to play with stuff like that.

Somebody, please!

Correct but also it’s addressable and doable f.ex. you can have those in 1Password and use their CLI tool to fetch secrets at runtime; it’s slightly clunky and takes a little time to set up but it works and doesn’t compromise prod security as long as the dev itself is not an idiot.

Yeah same. Whoever imagined that’s a good way to do things should be shot in public.

It’s not about developers leaking credentials from their poor management of tokens locally, it’s about devs not having access to those credentials. For example you are pushing to dockerhub with a corporate token, do you really want anyone but the CI server to have access to that token?

Ah, that’s true, sure you don’t want that – my point was that you can untangle logistical stuff like pushing to Docker registries on successful run from the actual linting script. On your machine you only want to run the latter.

Well, this really doesn’t solve much, you basically will have parts of scripts that you cannot run locally, hence again pushing continuously to repo to check if they work.

If you want to avoid YAML, isn’t dagger the thing you want?
They even have an Elixir SDK.
I’ve never tried it because the CIs I set up are usually so simple that it’s not worth the effort to learn how dagger works.
I actually don’t dislike YAML for CI and I also prefer Gitlab’s CI config in comparison to e.g. GitHub actions.

Regarding the original question, there is act which claims that you can run GitHub actions locally with it, so that might be something for you.

If anyone of you tries dagger or act, please share your experience.