dli
Hey everyone,
I have a GenStage producer that receives real-time events from a 3rd party websocket API and produces GenStage events. I want to process, then broadcast these events via a Phoenix channel and persist them to the DB as well.
For the latter, I wonder whether there is a way to both:
- Create chunks with a max size, e.g. ~20,000 events
- Create chunks of smaller sizes when a timeout occurs before the chunk is “full”
Both types of chunks should be persisted immediately.
I already have a working GenStage consumer that uses Process.send_after/4 to schedule a flush when the first event of a chunk is processed. However, I hope that a stream-based implementation could be terser.
Is this at all possible with Streams? AFAIK, Stream.interval/1 blocks the caller and is therefore not an option + I’d need to receive these events inside Stream.chunk_while/4.
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #blog-post
- #elixir-ls
- #ai
- #elixirconf-us
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 4- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
LostKobrakai
Broadway has batching capabilities by amount and timeout.
dli
IIUC, you’re referring to the
batch_sizeandbatch_timeoutoptions.Right now, Broadway would add too much complexity to the project. I’d like to avoid adding services like Redis or Kafka just to get some convenience for batching events.
WDYT, is this feasible with Streams? In the meanwhile, I also found Flow.Window but am not sure how to combine periodic windows and count windows.
LostKobrakai
Broadway can consume genstage producers. You seem to already be using genstage, so it shouldn‘t be a large change.
dli
Gotcha. Just curious, how would you solve it without Broadway?