Consuming streaming data

Questions / Help
,
1

hi!!

I’d like to practice using GenStage as a data pipeline sort of tool and I have a general question about how someone would approach the following scenario and if this is something elixir is suited for

If, for example, I wanted to continuously make HTTP calls to retrieve data from https://www.pubnub.com/docs/console?sub=sub-c-5f1b7c8e-fbee-11e3-aa40-02ee2ddab7fe&pub=demo&channel=pubnub-sensor-network and maybe do some transformations on it and then just output it to the console, how would you go about doing this with say, HTTPoison. Is this what Phoenix Channels are intended for?

I’m not sure if that example that I linked allows for making HTTP requests to the endpoint but let’s say I already had some data endpoint that allowed me to make a curl request ie:
curl https://www.streamingdata.com:port/
and just started pumping out data to my terminal, how would I then do this with HTTPoison and/or is there a better way to do it?

I’m hesitant to post my current approach since it seems incorrect and doesn’t actually work lol so was just wondering if anyone has done anything like this

thank you for your time!

2

Hi @blksheephw :wave:

Given the scenario you’ve laid out, it sounds like you’d want to create a GenStage producer that responds to demand by fetching data from your streaming data endpoint (via HTTPoison). Then, you could make a consumer that subscribes to your producer. That consumer would be responsible for puts-ing the data to your console.

FWIW, I think the BroadwaySQS producer code is a good example of how to build a producer that does a similar thing (it makes an HTTP request to Amazon’s SQS service).

(I’ve only got a few minutes at the moment, but I hope this points you in the right direction)

3 Likes