alvises
GenStage and twitter stream
Hi guys,
I’m playing a bit with GenStage, after reading the blog post on elixir with a full case description, and watching the youtube Jose’s speech.
I’m playing with it using twitter streams. The producer reads the twitter streams, receiving all the tweets linked to the hashtag #rio2016, and pass them to the consumers.
in the gist there are app, producer, consumer and log output
I’ve started 3 consumers. Once a consumer receives a bunch of tweets from the producer, I put it to sleep for 10s to simulate the processing. The problem is that the other 2 consumers don’t receive anything..
if I change the stream to a local big file, all the 3 consumers receive events…
Does anyone know why ?
Thanks
Alvise
First Post!
benwilson512
You haven’t overridden the default demand quantities. GenStage operates in batches, and the min / max batch sizes are 500 / 1000 by default, so unless you have more than that it isn’t gonna use more than one worker.
This is a good example of Default demand should be 1 · Issue #72 · elixir-lang/gen_stage · GitHub
Most Liked
mario
When my GenStage consumer(s) are too slow to handle the amount of incoming data (Tweets) the producer will stop ack the HTTP/TCP connection packages. The Twitter streaming endpoint will stop sending more data through the connection.
This is what whe want and follows the same pattern GenStage and Flow follow to consume data.
Without back-pressure, my producer would have to buffer the incoming Tweets or it’s internal message box would grow indefinitly. In both cases this would result in taking more and more memory, crashing the VM in the long term.
mario
I wrote a Twitter library from scratch (at that time, it was my first project to get startet with Elixir). It uses HTTPoison (wrapper for :hackney).
A few weeks ago, I’ve updated the streaming part of the lib to use GenStage.
You can have a look at the implementation here: twittex/lib/twittex/client/stream.ex at master · redrabbit/twittex · GitHub
Basically, the Twittex.Client.Stream receives incoming HTTP chunks from :hackney applying back-pressure at the TCP level with :hackney.stream_next/1.
The next thing I want to try out is using Flow and Flow.Window to implement features such as peak detection.
josevalim
It is a bug in ExTwitter. The stream is discarding messages from the process inbox, which is a big no-no. This means that the subscriptions for the second and third stages are never consumed:
https://github.com/parroty/extwitter/blob/master/lib/extwitter/api/streaming.ex#L110-L111
ExTwitter should use a reference and make sure it only consumes messages it knows about. This must be reported as a bug.
Ideally ExTwitter would provide a GenStage producer, to avoid hijacking the process inbox, which won’t work with a GenStage since both GenStage and ExTwitter are trying to use the same process inbox. You can check the issues tracker so see if someone already requested this feature.
Last Post!
demem123
Hi! I was trying out the twittex from the blog. But I got stuck on authorization error. I have set the config
config :twittex,
consumer_key: "xxx",
consumer_secret: "xxxx",
token: "xxx",
token_secret: "xxx"
but it keeps getting stuck when it tries to do a post to the https://stream.twitter.com from the repo. =(
I want to learn to create a streaming api client. =(
Thanks!
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"