ElixirConf 2017 - Scaling up Elixir at TpT - by @shanti
Introducing a new language into an organization with a large existing codebase is a major effort. Teachers Pay Teachers recently made that transition from PHP to Elixir and NodeJS last year, and I’ll share the process, the struggles, and the triumphs we experienced, and the lessons we learned.
We’re currently building on an internally-managed Kubernetes cluster, and we have multiple pods housing our API servers, which are all written in Elixir. The client-facing code executes on JS in the Browser + on Node.js workers and hits the API via intra-cluster RPC calls. We run with 8-12 pods on normal daily traffic, spiking to dozens during sale days. For the most part the servers themselves are somewhat-stateless, and we use MySQL as our primary data store. We also use message queues backed by Redis. The legacy system’s PHP codebase and Memcache sometimes access but are never accessed by the API.
We’re not using any distributed erlang facilities right now. If we had to set up a distributed lock manager, we’d probably rely on either Kubernetes’ built-in etcd instance or create a ZK instance. In-memory caches would be a strong contender for distribution within our API cluster, but we have no current need to consider those approaches.
In both cases, the additional operational overhead is a major factor against adopting more involved distributed architectures.
