gutschilla
CI/CD: What's the fastest CPU for Elixir compilation?
Hi,
I want to minimize the time required for the most-costly compilation step in our CI/CD environment.
Right now, We’re on GitLab using their runners to be installed on own machines. I tried with Kubernetes clusters on GCP and the Gitlab-provided runners and while this all runs fine and scales nicely horizontally, I am looking for quick build and test times. In our setup, the build step (fetching deps, comiling deps and own sources) takes the most time: a few minutes, depending on the project.
So I am looking into dedicated hardware for our gitlab runners where the major workload will be mix compile 
. Since the team isn’t big I don’t really need all the auto-scaling a Kubenetes cluster has to offer. I want quick result for individual builds from scratch.
Should I go many cores or for high single-core performance? Other than that I’ll go for local SSD storage and plenty of RAM (64GB).
Thanks for your insights!
Marked As Solved
dimitarvp
In terms of CPUs, I found the AMD Threadripper series quite good for compilation and overall dev work. Best results so far however I achieved with recent Xeon CPUs. They have bigger L1 / L2 / L3 caches than most consumer CPUs and I found them to be extremely adequate for compiling.
I’d say go for a Threadripper or a Xeon with at least 8 physical cores and a NVMe / PCI SSD (> 2 GB/s read/write speed). RAM isn’t of huge consequence for a compilation / CI machine. Even 16GB would be an overkill, unless the machine is also a dev machine and I misread you.
Also Liked
benwilson512
The biggest thing when making a CI/CD fast is minimizing the work that you need to do each time. If you’re having to compile all the dependencies from scratch every time, the biggest win will come from finding a way to cache that, regardless of what CPUs are available. We have a large project, but the actual compilation step takes < 10 seconds in our CI unless we are changing a dependency.
sorentwo
Tiered caches will help with the build time a lot. On CircleCI, even with small test runners, you are able to cache the hex deps, the compiled code, and PLT files if dialyzer is your thing.
I do this for all my applications, but here is an example for an open source project:
https://github.com/sorentwo/oban/blob/master/.circleci/config.yml
benwilson512
Fair enough! The Elixir compiler is parallel, so you should be able to benefit from multiple cores. SSD speed is of course critical. 64gb of ram is wildly more than enough.
Last Post!
dimitarvp
Yep, I’ve heard the dependency graph can severely impact the parallelisation of compilation
and it makes sense.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"