Messaging With Limits: Concurrent, Multi-Stage Data Processing in the Real World

[ElixirConf]

Messaging With Limits: Concurrent, Multi-Stage Data Processing in the Real World

Messaging With Limits: Concurrent, Multi-Stage Data Processing in the Real World

ElixirConf US 2024 - Brian Meeker

Welcome to ElixirConf

Our team at GetThru found ourselves with a scaling problem. Working with our existing infrastructure for sending text messages was becoming increasingly complex. Years of bewildering external requirements had led us to the point where the existing system was both incorrect and difficult to reason about.

In this session, we’ll go over a case study of our new concurrent, multi-stage data processing pipeline. Starting from the fundamentals, we’ll see how our team efficiently handled our existing external requirements and built in flexibility for the future. Finally, we’ll look at how we did all of this in our existing monolith, avoiding the complexity of microservices.

Comments welcome! View the elixirconf tag for more ElixirConf talks!