Data processing sans dependencies - Exir Saxby

[ElixirConf]

Data processing sans dependenciesing sans dependencies Exir Saxby

Data processing sans dependenciesing sans dependencies Exir Saxby

ElixirConf US 2024 - Eric Saxby

Welcome to ElixirConf

I have worked on numerous teams where teammates (or bosses) have pushed strongly for using specific libraries or frameworks, not because of best fit for our problems but because the proposed library was similar to something they’d used at previous companies using different languages.

I would like to live in a world where Elixir developers reach for OTP first, and then add libraries based on their actual needs.

Audience:

The main audience that I am targeting are developers who have come to Elixir from other languages where scheduling asynchronous work is difficult, and who expect that they must add a complicated framework in order to solve their problems.

Additional notes:

This talk is sourced from two main domains, US healthcare data and carbon emissions from global shipping logistics. Along the way I will try to give specific examples of real problems.

Points that I’d like to touch on:

• Separating the modular/functional domain from scheduling.
• spawn - Task - Task.Supervisor - GenServer - Supervision tree
• Enum - Stream - Flow - GenStage
• Registries.
• Patterns for domain-specific data models to facilitate data pipelines.
• Architecting from the runtime requirements and limitations of different parts of the work.
• How do Elixir built-ins scale across schedulers? How do they scale across VMs?
• Using OTP to limit requests to external systems (either rate limiting or throughput limiting), as well as back pressure.
• Deployments and graceful shutdown.

Comments welcome! View the elixirconf tag for more ElixirConf talks!