Using Process.send_after() outside of a GenServer?

Questions / Help
1 
defmodule APIResponseProcessor do
  defp update_log(api_url, error_message, {sleep_time, interval}, line_number, attempt_count) do
    log_level =
      case attempt_count < 5 do
        true -> :log_only
        false -> :warning
      end

    LogBook.main(
      "Error: #{inspect(error_message)} (attempt #{attempt_count}) - Retried #{inspect(api_url)} after #{sleep_time} #{interval} . . . ",
      __MODULE__,
      line_number,
      log_level
    )

    {:update_log, :ok}
  end

  defp retry_api_url(api_url, error_message, attempt_count, connection_type)
       when attempt_count <= 29 do
    with {:calculate, delay} <- calculate_delay(attempt_count),
         {:ok, randomized_sleep_interval} <- MiscScripts.random_sleep(delay),
         {:update_log, :ok} <-
           update_log(
             api_url,
             error_message,
             {randomized_sleep_interval / 1000, "seconds"},
             38,
             attempt_count
           ) do
      ApiInterface.connect_to_api(api_url, connection_type, attempt_count)
    else
      glitch ->
        ExceptionsHandler.raise_erroneous_value_alert(glitch, __MODULE__, __ENV__.function)
    end
  end

  defp retry_api_url(_, error_message, attempt_count, _),
    do: raise("Error after #{attempt_count} failed attempts: #{inspect(error_message)}")

  defp calculate_delay(attempt_count) do
    delay =
      case attempt_count do
        1 -> 200
        2 -> 500
        _ -> 1500
      end

    {:calculate, delay}
  end

  def main(
        {:error, %HTTPoison.Error{id: nil, reason: _} = error_message},
        api_url,
        attempt_count,
        connection_type
      ),
      do: retry_api_url(api_url, error_message, attempt_count + 1, connection_type)

  def main(
        {:ok, response = %HTTPoison.Response{status_code: status_code}},
        api_url,
        attempt_count,
        connection_type
      ) do
    case status_code do
      code when code in [200, 301, 404] ->
        {:api_response_processor, response}

      code when code in [400, 429, 502, 503, 520] ->
        error_message = status_code_to_error(code)
        retry_api_url(api_url, error_message, attempt_count + 1, connection_type)

      _ ->
        {:api_response_error, "Unexpected status code: #{status_code}"}
    end
  end

  defp status_code_to_error(error_code) when is_integer(error_code) do
    case error_code do
      400 -> :bad_request_error
      429 -> :too_many_requests
      502 -> :bad_gateway
      503 -> :service_unavailable
      520 -> :no_data_received
    end
  end
end

defmodule MiscScripts do
  @spec random_sleep(integer) :: {:ok, integer}
  def random_sleep(max_interval) do
    random_interval = :rand.uniform(max_interval)
    Process.sleep(random_interval)
    {:ok, random_interval}
  end
end

The api retry function uses Process.sleep() for the delay but I’d like to use Process.send_after() since this seems to be the best practice. All the information I’ve come across suggests that send_after() can only be used within a GenServer which seems like overkill. So, two questions:

  1. How, if possible, can Process.send_after() be used outside of a GenServer?
  2. Is using a GenServer for this really overkill or no?
2

I would answer your second question ‘no’. Tracking state is a sound reason to reach for GenServer. The main anti-pattern around GenServer is to use them for code organization. Conversely, docs say:

Use processes only to model runtime properties, such as mutable state, concurrency and failures, never for code organization.

which clearly applies to your case: GenServer — Elixir v1.12.3

That said, have you seen HTTPoison.Retry — httpoison_retry v1.1.0 I think there may also be http libraries with retry support built in…

3 Likes
3

The bit you’re missing is why people suggest to use send_after in a GenServer. The reason is that in a GenServer you don’t want to block the process, because that prevents it from handling other messages. It also already has a receive loop going, so when you get the message there is code to handle it.

If you instead have just linear code though and not a GenServer, how would send_after work? Your code is already written to block for the amount of time it’s going to sleep. Doing

Process.send_after(self(), :try_again, 5_000)

receive do
  :try_again -> :ok
end

is exactly as blocking as just doing Process.sleep(5_000) in the first place.

3 Likes
4

Having delved into different strategies utilizing a GenServer, it appears that incorporating a handle_call() callback is necessary. This callback, akin to Process.sleep(), operates in a blocking manner. Nonetheless, by encapsulating the handle_call() within a :poolboy.transaction(), the blocking aspect becomes inconsequential, as the tasks can be allocated across available workers. Is there anything I might be overlooking?

5

What benefit does this gain you? The calling process still blocks, and now you’ve got a genserver and poolboy added to the mix which only complicates things further.

1 Like
6

I was pondering the same myself. It appears that, given how both Process.sleep() and handle_call() block execution, it might be a more beneficial approach to structure the GenServer around handle_info() callbacks, utilizing Process.send_after() for the retry delay. This way, we can effectively harness OTP’s asynchronous processing capabilities while keeping the system’s complexity in check. Is there anything else I might be overlooking?

7

Hey Maxx, the real question to ask is, where does this APIResponseProcessor code run? The whole genserver or no genserver thing is a red herring if APIResponseProcessor is just running in some linear code somewhere.

8

The goal is to create a versatile interface that enables multiple modules to concurrently invoke various APIs while efficiently handling their individual responses. Additionally, it should include a resilient retry mechanism to tackle any availability-related issues.

9

The question though here isn’t really about modules it’s about processes and what those processes should experience when they call this code. If I have a process and it calls a function that needs to make an API call, and the API call needs to retry, what do you want to have happen in the function? If the answer is “block” then Process.sleep does the job.

1 Like
10

Well, there’s a specific asynchronous task dedicated to fetching real-time price data from a cryptocurrency exchange’s API for multiple trading pairs. Given the critical nature of this data to ensure the smooth flow of subsequent operations, it appears justifiable to opt for a synchronous approach in this context. This becomes particularly relevant when considering the possibility of retries due to initial API call failures caused by temporary availability issues. That said, are there alternative design patterns worth considering for addressing this situation?

11

This sounds like the perfect candidate to just emit data to a Phoenix.PubSub and have consumer(s) attached to it.

Now if you want a module that does something generalized and you can just pass a remote data provider to it (as you alluded to) then yes, it’s doable and from a quick skim of your code you are close to that goal.

Or you can just plug something like Retry and call it a day.

1 Like