AstonJ
ElixirConf 2019: Build Efficient Data Processing Pipelines with Broadway - Marlus Saraiva
Announced in February 2019, Broadway is a new open source tool developed by Plataformatec that aims to streamline data processing pipelines with Elixir. It allows developers to build complex GenStage topologies that can consume and process data efficiently from different sources, including Amazon SQS and RabbitMQ. In this talk, we’ll discuss some of the main concepts behind Broadway, how it leverages OTP to achieve fault-tolerance, sharing implementation details and architectural aspects. You’ll learn how to build efficient data processing pipelines and how to optimize them using real-time metrics based on Telemetry events.
By @msaraiva
See all threads tagged with: broadway, telemetry
View all Elixir Conf 2019 threads: elixirconf2019
Most Liked
msaraiva
Hi @dantodor!
The project is currently not open source. I have plans to make it so in the near feature but before I can do that, I need to do a couple of things:
-
Push the necessary changes I made to Broadway upstream. This will require reviewing/validating the set of event/metrics chosen as well as the implementation, tests and docs.
-
Publish a still private experimental library I’m working on that allows creating reusable components in Phoenix/LiveView (something similar to
ReactorVue.jsbut 100% Elixir). This library was the one mentioned by @chrismccord at the end of his keynote at ElixirConf when he was asked about having components in Phoenix. I’m using the dashboard project to validate some of the concepts behind it and since it’s still a PoC, the API is changing on a daily basis.
So, I’m really excited about those projects and I agree with you when you say that it would be the perfect tutorial for both technologies. I’ll write a blog post about it explaining the details as soon as I get all those exciting things stable enough for a first release.
Cheers.
-marlus
msaraiva
Hi @eaverdeja!
The CPU History graph was shamelessly copied/adapted from @sasajuric’s awesome demo_system, so all credits to him. You can find the graph code here and a video of the graph in action here.
As for the CPU data, I’m using :cpu_sup.util([:per_cpu]) to retrieve individual core usage. Pay attention that in Saša’s demo he’s using the schedules usage instead of the system CPU usage. Which one you should use will depend on your requirements.
Popular in Talks
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Move fast and fix things! Error tracking with less noise for Elixir devs who love to ship. Start your free dev account.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"