Genstage + polling

Questions / Help
1

Hi guys!

I have been working on genstage producer + polling mechanism that gets data from an external source using

def init(initial_demand) do
  {:producer, initial_demand}
end

def handle_demand(demand, state) do
  send(__MODULE__, :work)
  {:noreply, [], demand + state}
end

def handle_info(:work, demand) do
  events = <get data>
  Process.send_after(__MODULE__, :work, 60_000)
  {:noreply, events, demand - Enum.count(events)
end

its all good but then I noticed that <get_data> gets called multiple times with the same state a) when it gets new demand from the consumer it calls send_after if demand < min_demand and b) when it returns from the send_after from processing the previous demand. I feel its redundant for the same action with the same state to be called multiple times. :frowning: Is there a better way of doing this?

Thanks! =)

2

Send the :work message from init instead of handle_demand

That way you’re effectively setting up a tail recursive polling loop, just with message sending rather than function calls - while still allowing the messages that drive handle_demand to be received and do their thing.

We do something similar in an app we run

1 Like
3

Hi,

Thanks for the reply! I’ve been stumped! I feel like I’m just missing one small piece. =(

I was thinking about this. If I sent a message on init then it will do the job every interval, i still need to call that work on demand though to get the events. Is this what you meant?

  schedule_job()
  {:producer, initial_demand}
end

defp schedule_job() do 
Process.send_after(self(), :work, ....)
end

def handle_demand(demand, state) do
     send(__MODULE__, :work)
    {:noreply, events, demand + state}
end

def handle_info(:work, state) do
  events = <get data>
  {:noreply, events, demand - Enum.count(events)
end

thanks!

4

I have the same issue but instead of polling, I need to listen to a websocket of external resource. It means I have to listen 24/7 to get real-time data from external resource. Is there a way to that? (I’m new to Elixir).
Thanks

5

hi,

I’m also new-ish to elixir but I saw this link Use Phoenix to parse content on external websocket Is this something that can help?

You can probably just send a message to the process/application that will be listening and tell it to run a function that gets the data every 1 second.

Good luck! =)

1 Like