Tacnoman

Tacnoman

How can I do a batch process requests

Hello guys
I must create an api with a great performance and I want to create with Elixir
I have a process (slow) that I must run after some requests. I want to make this flow

In each request, save the data received in memory
After x requests, send to another api (or after x seconds)

In node I can make this:

let batchData = []
const handlerRequest = (req, res) => {
  batchData.push(req. body.data)
  if (batchData > 1000) {
    // Process to send to another api
    batchData = []
  }
  res.json({ success: true })
}

Or

let batchData = []
setInterval(() => {
  if (batchData > 1000) {
    // Process to send to another api
    batchData = []
  }
}, 10000)

const handlerRequest = (req, res) => {
  batchData.push(req. body.data)
  res.json({ success: true })
}

How can I do something like this in Elixir Phoenix?

Thanks for this

Marked As Solved

kokolegorille

kokolegorille

You can do this with OTP. Here is a simple GenServer that send itself a regular tick, and check the size of it’s state when data is pushed.

defmodule Demo.Gs do
  use GenServer
  @interval 10_000
  @initial_state <<>>
  
  def start_link(arg \\ @initial_state) do
    GenServer.start_link(__MODULE__, arg, name: __MODULE__)
  end

  def push(data) do
    GenServer.cast(__MODULE__, {:push, data})
  end
  
  def init(arg) do
    Process.send_after(self(), :tick, @interval)
    {:ok, arg}
  end
  
  def handle_cast({:push, data}, state) do
    state = data <> state
    case String.length(state) > 1_000 do
      true -> 
        process(state)
        {:noreply, @initial_state}
      false ->
        {:noreply, state}
    end
  end
  
  def handle_info(:tick, state) do
    process(state)
    Process.send_after(self(), :tick, @interval)
    {:noreply, @initial_state}
  end
  
  defp process(state) do
    IO.puts state
  end
end

It is just a simple example, just printing state after 10 seconds, or when pushed data is bigger than 1_000

PS. You just need to change process() with your own implementation.

Also Liked

Tacnoman

Tacnoman

Really, really thanks for this

My final solution:

defmodule Speed2.BatchProcess do
  use GenServer
  @initial_state []
  @interval 10000

  def start_link do
    GenServer.start_link(__MODULE__, [])
  end

  def init(initial_data) do
    Process.send_after(self(), :tick, @interval)
    {:ok, initial_data}
  end

  def handle_info(:tick, state) do
    process(state)
    Process.send_after(self(), :tick, @interval)
    {:noreply, @initial_state}
  end

  def get_my_state(process_id) do
    GenServer.call(process_id, :get_the_state)
  end

  def push(process_id, value) do
    GenServer.call(process_id, {:push, value})
  end

  def clean(process_id) do
    GenServer.call(process_id, :clean)
  end

  def process(state) do
    IO.inspect state
  end

  def handle_call(:get_the_state, _from, my_state) do
    {:reply, my_state, my_state}
  end

  def handle_call({:push, value}, _from, my_state) do
    new_state = my_state ++ [value]
    case Enum.count(new_state) > 10 do
      true -> 
        process(new_state)
        {:reply, new_state, @initial_state}
      false ->
        {:reply, new_state, new_state}
    end
  end

  def handle_call(:clean, _from, my_state) do
    {:reply, my_state, @initial_state}
  end
end

Where Next?

Popular in Questions Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
johnnyicon
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
nobody
How to bind a phoenix app to a specific ip address? could not find anything about that, nowhere, unfortunately, but for me this is quite...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

Other popular topics Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43622 214
New
Lily
In templates/appointment/index.html.eex: &lt;%= for appointment &lt;- @appointments do %&gt; &lt;tr&gt; &lt;td&gt;&lt;%= appoi...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
marick
I had some trouble figuring out how to make many-to-many associations work. Once I got it working, I wrote a blog post. Because I’m a nov...
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New

We're in Beta

About us Mission Statement