Jenkins is a CI software that you can deploy in-house for free. Jenkins has a ton of problems, the most serious of them being that it has over 500 open issues (many of them security related) and its interface is from the 90’s.
Now, given you can fix some of this issues with plugins, a basic rule applies with Jenkins: the more plugins you have, the more security vulnerabilities you have.
Looking for an alternative
So, after using Jenkins for 2 months and having being attacked each time we got the service up (it’s a miracle Google dind’t ban us yet for “mining bitcoins”) we have decided to look for alternatives.
Yes, I know you can put Jenkins behind a super duper safe private network, but we don’t have that yet. Nor we will have it anytime soon, so we need a safer alternative.
Open Source CI tools
Now, there are a myriad of lists out there about open source alternatives to jenkins, the most common ones being (in no specific order):
BuildBot
CruiserControl
TravisCI
Go
But we would like to know, what you use and why. Also, a few questions:
Is it secure (can it be publicly exposed)?
Is it easy to install and setup in a local environment?
Does it have good documentation and support (even courses, no matter if payed)?
Is it really open source (can we fix bugs if we find them)?
Let me know your opinions and personal experiences!
One way to go is to make your own ci from scratch using a bunch of services from gce, “serverless” functions to handle notifications from gitlab/github, spot instances for cheap compute (started and stopped on demand, just to run the tests), container builder for, well, building containers, and then something like docker-compose to run the tests …
I do use GitLab (the service) for my personal staff for some years now, in the company I’m working for, we use a self hosted GitLab and we use it for CI/CD as well.
In the company its not publicly exposed though, and to be honest, if you do confidential stuff, you should do so as well, independently of the service you use for CI/CD. Even those that might seem to be secure, might have undiscovered vulnerabilities which again might lead to leaking source code, keys, passwords or other kinds of secrets.
You really shouldn’t even think about self hosting before you are able to properly secure the network that it is hosted in…
Why not just use GCB for everything? Building containers can be used to model your test running, linting, etc. We do this and it works well. The servers are very fast. And it can be hooked into GitHub natively now, so PRs can get failure notifications. We have an external service to cancel builds on repeat pushes, but that shouldn’t be strictly necessary.
I have already used that plugin but I can’t say I am a fan of the huge amount of clutter that comes with it. BlueOcean alone installs over 30 plugins iirc. You won’t ever use half of them, and even then the BlueOcean plugin only gives you but a partial skin to some areas of Jenkins, not all. In the end you still need to know how the basic jenkins UI works without the skin because you will end up using it.
Have you used any of the alternatives you mentioned? What is your opinion?
You can use shell runners as well, but I really do not like those. If you use them, you basically have Jenkins with YAML instead of groovy, and it still puts the burden to the systems maintainer to provide and update all the necessary tools that are necessary to run the tests and deployments.
Docker runner have the benefit, that they provide an easy way to run the same tests with multiple versions without needing to juggle with asdf or any other kind of environment manager. Just swap out the container… No need to manually provide a postgres database… Just add the service: postgres:10 to your CI description.
As a user I really like docker-based runners. Our ops team started to like them as well, after a lot less tickets were opened because of the environment does not match the requirements anymore when we switched from Jenkins to Gitlab-CI with docker runners.
I’m using GitLab CI for a bunch of personal projects on top of GKE using their provided Helm chart including the Kubernetes-based runner. At work we use Jenkins (LTS version) with the Kubernetes plugin. This is the first position where I’m not responsible for maintaining/supporting the Jenkins footprint, just contributing Jenkinsfile content and shared Groovy libraries. Neither is a flawless/frictionless experience but they’re both able to let me get my work done, which is all I actually care about.
Jenkins is definitely a “nobody gets fired for choosing it” choice and there’s an immense body of prior art in the form of plugins, writing, community knowledge, etc. which aren’t negligible resources compared to some of the alternatives. They’re struggling to reinvent themselves in a space with rapidly increasing competition and demands to become “more cloud-native”. They’re making attempts in that space with some “serverless” initiatives (one-time disposable master) and their Jenkins X sub-project, which I found intriguing but ultimately way too opinionated in the wrong direction for my personal and professional circumstances. Others may find it more palatable.
Ultimately I think the most damning thing I can say about Jenkins is that, even with a whole fleet of plugins, the only programming languages that feel like first-class citizens are those that run on the JVM, which makes total sense if you know the project’s history and its own tech stack.
Hear hear!
I’m rather happy with Google Cloud Build as a component but I it doesn’t have anything approaching feature parity with an actual CI platform like Jenkins or the other alternatives presented in this thread. I like it as a way to not provide insecure access to host Docker daemons on my Kubernetes cluster, and to avoid Docker-in-Docker. Google just shipped an opt-in mode for GCB to build images using Kaniko, which should help build times a lot for some users.
Are the operator of the Jenkins install? If not, don’t trouble yourself whatsoever with how the functionality is packaged and focus on whether or not its behavior is actually useful to you, or is holding your back. I agree with complaints that Blue Ocean doesn’t go far enough, and it’s frustrating that you can get different forms of information presented at drastically different levels of quality by switching between the classic and Blue Ocean UIs for a given build.
Miscellaneous remarks about some of the other tools mentioned in this thread:
Drone just shipped an attempt at better native Kubernetes integration in the last few months, and that’s about all I know about it. Some friends are eager to try it out soon.
Concourse is philosophically wonderful and suits my personal temperament the most, but it is a really, really hard sell to some organizations that they may only reconfigure the CI behavior through code files and a CLI. This is especially true for orgs where they have non-technical stakeholders that like to peek in or maybe kick off a predefined job here and there. There are also some performance concerns when running Concourse and its agents both in Docker, because of their inner containerization/isolation layer.
I run Hydra at home for some Nix packages I maintain as part of my home-manager config, but it’s a labor of love and not something that is particularly practical to adopt without doing a bottom-up adoption of Nix throughout your entire tech stack. Otherwise you’re undergoing a bunch of toil and then the benefits stop at your CI phase without propagating into deployments and infrastructure. I’ll be eagerly watching a hosted Nix-based solution called Hercules CI to see if it makes the technology more accessible. There’s also a community project to make onboarding to Hydra way simpler, but I haven’t tried it out yet.
The Build subtool at Sourcehut (FKA sr.ht) supports a bunch of build environments and the full stack is open source, written in Python IIRC. Supports NixOS as a build environment!
Way, way, way out in the weeds are some of the attempts to create a Kubernetes-native solution that builds on the same roughly-DAG-driven model that Concourse uses. Knative Build is probably the one with the most momentum right now.
I suggest this because nothing mentioned in this thread, all solutions are great (gitlab ci is my favorite and using the shared runners is a great solution for CI-CD solution for start up projects)
You can use Alloy-CI, it’s a great solution for this problems and it’s written on Elixir
Personally i don’t use up the date but looks great to do experiments
Concourse has a standard resource specifically for updating the configuration of a pipeline from files inside of, say, a git repo, so that’s still a fine pattern. Starting a pipeline in the interface is as simple as being logged in, clicking it, and clicking the run button, I have a few like that to cause releases and version bumps and so forth that aren’t triggered by anything but me starting them.
Performance is a bit odd at times I’ve noticed, and the interface stops updating after a while in firefox, no clue why there, but overall I’ve been pretty happy with it. I have one public pipeline for a project I host for someone that I can show if anyone is curious how it works. It uses vault for the secrets, builds a java project, publishes it, sends announcements, etc… etc…
We’re building an open source one right now - buildex - extends what docker supports (adds volume mounts during build) - communicates over rabbitmq (so highly observable) - polls using tentacat do you don’t need to open network ports to callback hooks - supports GitHub oauth - written in elixir - runs in docker containers - very early days - contributions / external involvement very welcome -
Seeing the discussion here makes me think it’s a good place to ask a question – what CI would you guys recommend for an Elixir & Phoenix project with very little native compilation in the deps? We currently use Semaphore and I am underwhelmed by it; it’s extremely slow (I have a half-broken laptop with an i3 CPU that compiles deps and project 10-15 secs quicker!) and it doesn’t cache dependencies – at least our several half-hearted attempts to force it to do so have failed which tells me it’s not documented enough or is not flexible enough.
I am not finding it quick (or even useful) and would shoulder the burden of migrating to another CI. I am looking for a much faster compilation speed and easily configurable build steps caching (in this case we’d like to only cache deps but many other needs could arise in the future).
I doubt the business people would shut me down if the next CI costs $50 more a month so let’s for now assume the price isn’t a show-stopper.
We’ve used CircleCI professionally. Starting at USD $50 a month for two containers and unlimited builds. We haven’t really found the need to try out alternatives as it has covered all our needs. It tests, builds and deploys our pretty large umbrella app in five minutes.
Our app isn’t huge currently; ~350 files to compile. Yet with dependencies compiled every time the whole thing goes to about 5 minutes which is absolutely embarrassing. It’s too long.
For those of you using GitLab, do you all use Docker? I mean, do you need docker to use Gitlab?
No, it has a lot of bonuses if you’re working with k8s but it doesn’t require docker though I’ve found life is easier if you use docker agents to build your releases with it.
Though it’s probably a bad measure for compile time, our app is around the same size, ~360 files and another 1,000+ files for dependencies.
It takes one minute to compile our app without caching any dependencies. It’s pretty easy to cache artifacts though, and we could probably get it down to near no compile time like @sanswork.
We didn’t do very serious efforts so far – but still tried several times even after the official docs method didn’t help, and still failed with Semaphore.
