kanishka

kanishka

Gitpod instead of CI?

For a few years now, I have always been annoyed at how much effort it takes to tune caching in CI so that it can get close to the rebuild time that devs are used to in local development. At one job, I had developers just log into a dedicated ec2 for their branch and run builds, tests, and preview deploys by ssh’ing in and running commands. It was amazingly fast and I didn’t have to come up with and maintain complex CI caching rules. It was slightly economically inefficient, but the team was small.

Going back to my favorite setup, I wonder if anybody has tried trusting developers, forgoing CI, and just requesting everybody run the test suite inside of terminal in Gitpod before merging? I think Gitpod could be a nice compromise between my former ec2 setup and cheaper containerized environments, without having to maintain some complex caching rules in a CI workflow language.

Note: I have done most of this thinking at various jobs while I wait for CI to finish before merging my PR’s.

Most Liked

w0rd-driven

w0rd-driven

You may be interested in https://earthly.dev/. Several official packages in the Elixir/Phoenix ecosystem use it.

There are also things like dev containers (Create a Dev Container) that sets up local Docker containers for development and VSCode connects to the container. It’s not CI/CD specific but you can create local containers that more easily “graduate” to production environments because you’re using the same tooling locally.

I haven’t used earthly or dev containers beyond throwaway projects. Gitpod is absolutely an option, but I’m wondering what you’re looking for exactly. I use CI/CD for automated test suites and deployment. Developers can run tests locally but CI is a means of verifying we haven’t broken the build before deployment. If your team is small enough you can always trust them to not break things but as a developer on Laravel at the moment, I have CI in place because I can’t even trust myself. My team also primarily uses it for the automated deployment step as some of our older projects lack meaningful test coverage.

As the person on my team tuning caching for builds, I understand your pain points. The biggest bottleneck for us was the network transfer of OS and framework packages. “Baking in” OS updates and common framework packages to the container running on Gitlab CI reduced build times from 5 to 1 minute. That alone was a huge win considering how many jobs have executed in the years that has been in place. We’ve likely saved days, weeks, or months by now. Using a dedicated runner instead of the shared runners also greatly reduced wait times. We do cache framework packages but that’s done by branch so new feature branches take longer as there is no cache for the initial push/pipeline job.

Are you doing things like hosting a network-local instance of Hex with your common packages? That’ll reduce the time for that initial build and dependency download. There may be more in-depth caching strategies around building the container that can be a little less painful overall. If my team weren’t so pleased with Gitlab, we’d be looking at earthly or GitHub actions now that they’re rolling out dedicated runners there as well.

D4no0

D4no0

At my previous jobs I was using custom Gitlab docker runners, setting cache on them is as simple as adding a few lines in the config, you can even configure if you want to store cache on a remote location. I was running tests and credo styling on big projects (more than 40 dependencies and 200+ source files) in 30 seconds each, it takes longer for runner to start the docker environment than to run the pipeline. You can deploy your runners on any instance that is running linux, and since the system is using a pull system you don’t have to worry about any proxy configurations.

The advantage of using a full-fledged CI is that you can also run integration tests, witch is very important in a lot of projects.

Supamic

Supamic

All of the positives of CI are worth the effort in my experience but being able to add a bit of Gitpod config to my repo (even just on a branch) and then build the project by appending the repo URL to the Gitpod URL, like so gitpod.io/#github.com/user/project for quick launch is certainly handy! Especially when spinning up other people’s public projects since I just need to fork, add my Gitpod config files for Elixir to the project and type in the new URL for launch.

Previously, I found the build process for Elixir on Gitpod problematic, in that it took too long to rebuild and install and the timeout was 5 min on the free plan so it became annoying if I was just reading some documentation in between bug fixing and had to wait 5 minutes to restart the workspace. The install process I was using also stopped working recently because Erlang Solutions stopped hosting the latest Elixir 1.14 packages for Ubuntu 22.04, which Gitpod uses as a base image. The main reason this is a headache remover for me is that I work in a Post-secondary institution which means we get Windows computers to work locally and while all my deployments for work are on various Linux-flavoured systems, managing multiple versions of Elixir on Windows OS is excruciating and to be avoided at all cost so Gitpod becomes an easy to use and delete sandbox environment that I don’t maintain and more importantly no need to convince my managers to provision and resource on our systems or in the cloud.

So through some perseverance over the last couple of days, I was able to write some config scripts which does the lion’s share of installing Erlang and Elixir using ASDF (which I had never been able to get working before) in a temporary Docker image which saves significant time on workspace restarts because Gitpod caches these and as long as you don’t update the customized .gitpod.Dockerfile, it will pull them back down in 30 seconds or so. I also use the “gitpod/postgres” base image to build upon, which persists the database info between restarts, and so that it is ready for Phoenix Ecto to use. Since your workspace files (files under the Git repo file structure) are saved between restarts, the compiled dependencies only need to be compiled the first time as well. This setup has allowed me to take a large project which could take 10-15 minutes on initial build and restart it any time in less than 2 minutes with the latest repo commits, which of course is the goal of Gitpod and without this advantage wastes more time that you gain.

I made a github gist of my 3 scripts which work in tandem and you can certainly strip them down to be more streamlined for your purposes but I wanted them to work with new and old Elixir projects without having to rejig so they have some backward compatible pieces and some redundancies. Please use, comment and fork publicly for wider community usage.

TL;DR
Elixir (for Phoenix w/ Postgres) Deployment on Gitpod using ASDF and Dockerfile

Where Next?

Popular in Discussions Top

blackode
Elixir Upgrading is so Simple in Ubuntu and It worked for me Ubuntu 16.04 git clone https://github.com/elixir-lang/elixir.git cd elixir...
New
Rustixir
Hi everyone, im working on find best language/framework/system for high concurrency, high performance and stable performance after wor...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New
Crowdhailer
I’ve been hearing much about the new formatter and it’s something I have been keen to try. I find examples buy far the most illuminating...
248 19204 150
New
mbenatti
Following https://github.com/tbrand/which_is_the_fastest |> https://raw.githubusercontent.com/tbrand/which_is_the_fastest/master/imgs...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31142 143
New
sergio
There’s a new TIOBE index report that came out that shows Elixir is still not in the top 50 used languages. It also goes on to call Elix...
New
rower687
Hi all, I’ve been reading a lot about the “let it crash” term and how supervising processes and the whole messaging passing make an elixi...
New
100phlecs
Are there any downsides, like perf issues, to putting all functional components in CoreComponents as long as you prefix it with the conte...
New
AstonJ
If so I (and hopefully others!) might have some tips for you :slight_smile: But first, please say which area you’re finding most challen...
New

Other popular topics Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 41539 114
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New

We're in Beta

About us Mission Statement