Announced in February 2019, Broadway is a new open source tool developed by Plataformatec that aims to streamline data processing pipelines with Elixir. It allows developers to build complex GenStage topologies that can consume and process data efficiently from different sources, including Amazon SQS and RabbitMQ. In this talk, we’ll discuss some of the main concepts behind Broadway, how it leverages OTP to achieve fault-tolerance, sharing implementation details and architectural aspects. You’ll learn how to build efficient data processing pipelines and how to optimize them using real-time metrics based on Telemetry events.
I watched over the weekend some of the videos from Elixirconf… While I liked most of them, I found the presentation of Broadway integrated with Liveview fantastic… As the author said nothing about the code, anyone has an idea if that code is published somewhere or not ? I think it’s the perfect tutorial for both technologies.
The project is currently not open source. I have plans to make it so in the near feature but before I can do that, I need to do a couple of things:
Push the necessary changes I made to Broadway upstream. This will require reviewing/validating the set of event/metrics chosen as well as the implementation, tests and docs.
Publish a still private experimental library I’m working on that allows creating reusable components in Phoenix/LiveView (something similar to React or Vue.js but 100% Elixir). This library was the one mentioned by @chrismccord at the end of his keynote at ElixirConf when he was asked about having components in Phoenix. I’m using the dashboard project to validate some of the concepts behind it and since it’s still a PoC, the API is changing on a daily basis.
So, I’m really excited about those projects and I agree with you when you say that it would be the perfect tutorial for both technologies. I’ll write a blog post about it explaining the details as soon as I get all those exciting things stable enough for a first release.
That is very good news, really looking forward to it… This is the thing I’ve been waiting for ages …
I’ve been playing around with Vue and with Elm, but I prefer 100 times to handle things on the backend…
If you ever feel you could use a helping hand to speed up things, please feel free to reach out
Thanks for the talk @msaraiva! Very cool stuff, clear explanation of a non-trivial supervision strategy, implementing the dashboard with LiveView, Telemetry integration - like wow. Seeing the stats gathered by Telemetry change by updating the topology in realtime is
I agree that all of it should render an awesome tutorial/blogpost/whatever.
In fact, I’m quite curious as to how you managed to update the CPU History graph while using LiveView. Could you share some insight as to how you got that to work?
The CPU History graph was shamelessly copied/adapted from @sasajuric’s awesome demo_system, so all credits to him. You can find the graph code here and a video of the graph in action here.
As for the CPU data, I’m using :cpu_sup.util([:per_cpu]) to retrieve individual core usage. Pay attention that in Saša’s demo he’s using the schedules usage instead of the system CPU usage. Which one you should use will depend on your requirements.
Have you published any of the code from the presentation yet?
I need to a graph for a business process and I an researching how to draw graphs from LiveView. Can you tell me what have you used?
…
