@nubunto I’ll share the story of how we ended up in a place where all new (internal) projects are being written in Elixir. The journey started about 18 months ago. By this time I had already given two or three internal tech talks about Elixir. My team was tasked with writing a service that would ultimately be used by a customer, though not directly. The flow looked like customer runs CLI thing => hits web service => web service asks us for authorization. Because the customer is ultimately waiting on our response, latency was a key issue. I realized this was my chance to broach adding Elixir to our environment.
I prototyped the service three times: once in two of our official tech stacks, and also in Elixir (using only Plug at this time since I believe Phoenix was in its relative infancy, and we didn’t need any UI). I then deployed each version to Heroku’s free tier and began to hammer on it from an EC2 machine in the same Availability Zone as the Heroku dynos. I then retested each with Heroku’s high performance dyno. One of the official tech stacks had difficulty meeting the (admittedly absurd) latency SLA at all. The other could meet the median and average portions of the latency SLA, but not the 90th percentile (due to global GC locks). Elixir on free Heroku tier exceeded the latency SLA. This was enough to get us discussing Elixir as a viable option.
We then weighed the relative business and engineering costs necessary to bring one of the other solutions up to meet the latency SLA (90th percentile of 12ms, including network transport time). Given our organizational competence in the other stacks, we opted to go with one of those, despite the greater engineering and business costs. But Elixir had its foot in the door!
A couple months later, I was informally discussing Elixir/Erlang with some SysOps folks, because I was having difficulty with an exrm-generated init-script to be fully Debian-compliant so I could manage an Elixir service via Puppet. SysOps’s reaction was basically, “Oh, Elixir is just Erlang. Oh, it’s a dependency-free tarball, and I don’t have worry about OS-package conflicts between it and, say, RabbitMQ. Oh, and we can manage its config via Puppet (via conform-generated .conf file). This is way easier for us to deploy and manage than either of our other two official stacks. Grassroots-approved!” I wrote plug_statsd so SysOps could monitor uniformly with other applications and began using fpm to generate OS packages instead of tarballs.
By this time I had also generated interest in the language with a couple other devs and a couple people in SysOps, assuaging the “It’s hard to find people” trope. With enough line folks in dev and SysOps on-board it was a relatively straightforward to jump to “All new (internal) projects in Elixir.”
We’re nearing a year in production for the first Elixir service, which has been great for me. The only time I’ve had to even think about the first production service is when an upstream vendor changed their not-SemVer API in a .z release
. I’ve even had to ask coworkers if they still use the service because I never hear any complaints (they do). As our team has grown, I have opted for growing Elixir talent rather than finding it, especially since that ultimately increases the size of the Elixir community.
We also have a full Continuous Deployment pipeline that builds releases, OS packages, publishes packages. Puppet configures with whatever credentials SysOps establishes (and they can change them without my involvement), deploys the latest packages, ensures the service is started. This uniformity makes staging => production super easy.