The One Billion Row Challenge in Elixir: From 12 Minutes to 25 Seconds - Raj Rajhans | Code BEAM

CodeSync · February 24, 2025, 2:00pm

This talk was recorded at Code BEAM Europe 2024. If you’re curious about our upcoming event, check https://codebebeameurope.com

The talk presents a case study on optimizing Elixir code for demanding tasks. The challenge is to read a file containing one billion lines of weather data and generate a report with minimum, average, and maximum temperatures for each station.

Through this talk, the audience will learn how to benchmark and profile Elixir code, how to model problems in order to make the best use of concurrency in Elixir, and implement performance-tuning strategies effectively. The talk unfolds as a story of performance transformation. Starting with code that initially took over 10 minutes to process the data, we’ll see how the runtime went down to just 25 seconds.

By the end of this talk, attendees will gain a deeper understanding of performance optimization in Elixir and leave equipped with practical tools and techniques to enhance their own applications.

Comments welcome! View the code-sync tag for more Code Sync talks!

sreyansjain · February 26, 2025, 2:57am

Amazing talk. Deep performance research work. Well done!

thiagomajesk · February 28, 2025, 12:08am

Indeed an amazing talk! Congrats @rajrajhans
I’ve used Flow before for similar tasks and now I’m super curious.