I need some help with external APIs

Questions / Help
1

Hey, guys! How are you?

I’m doing a challenge from a course about programming and choose to do it using Elixir since I want to learn how to create APIs using Phoenix…

I need to loop over 5 pages that return me a json with a bunch of numbers on each page, put them inside a json of my own with all the numbers ordered, that part I got already

But the problem is, there is catch in this challenge, sometimes the external API server will throw me a status 500 on purpose. How would I retry the same request that I did if I get back a status 500?

For now the part where I make the request is like this

def get_numbers(page) do
        "#{@base_url}#{page}"
        |> get()
        |> handle_get()

end

defp handle_get({:ok, %Tesla.Env{status: 200, body: body}}), do: {:ok, body}
defp handle_get({:ok, %Tesla.Env{status: 404}}), do: {:error, "Page not found!"}
defp handle_get({:ok, %Tesla.Env{status: 500}}), do: {:error, "Internal error"}
defp handle_get({:error, _reason} = error, _page), do: error

How would I retry the request if it hits my third handle_get until the status is 200 so it hits my first handle_get?

Also, I’m doing all the request recursively like this

def call() do
    page = 1

    page
    |> recursive_call()
end

defp recursive_call(page) do
    page
    |> Client.get_numbers()
    |> handle_response()

    page = page + 1

    if page <= 5 do
        recursive_call(page)
    end
end

defp handle_response({:ok, body}), do: {:ok, Number.build(body)}
defp handle_response({:error, _reason} = error), do: error

But I should be checking if the body that the api is returning my is empty, like with some pseudo code

If body == {“numbers”: []}:
stop recursion

2

The Client.get_numbers should have a way to retry in case of failure. You can code it yourself (in the function, pattern match on the response, in case of error call the function again), or you can use a library that abstracts it for you. For example GitHub - safwank/ElixirRetry: Simple Elixir macros for linear retry, exponential backoff and wait with composable delays

3

How would I do a pattern match on the response from the get() inside the pipe chain? Can I make something like

def get_numbers(page) do
        "#{@base_url}#{page}"
        |> get()
        case {:ok, Tesla.Env{status: 500} do
            get_numbers(page)
        end
        |> handle_get()
end
4

Maybe this is?

def get_numbers(page) do
    "#{@base_url}#{page}"
    |> get()
    |> verify_status(page)
    |> handle_get()

end

defp verify_status({:ok, %Tesla.Env{status: status}}, page) do
    if status == 500 do
        get("#{@base_url}#{page}")
    end
end

defp handle_get({:ok, %Tesla.Env{status: 200, body: body}}), do: {:ok, body}
defp handle_get({:ok, %Tesla.Env{status: 404}}), do: {:error, "Page not found!"}
defp handle_get({:error, _reason} = error), do: error

But now when I run the code I get an error no function clause matching
Called with 1 arguments

  1. nil

Attempted function clauses (showing 3 out of 3)

defp handle_get({:ok, %Tesla.Env{status: 200, body: body}})
defp handle_get({:ok, %Tesla.Env{status: 404}})
defp handle_get({:error, _reason} = error)

5

Changed the code again…

def get_numbers(page) do
        "#{@base_url}#{page}"
        |> get()
        |> case do
            {:ok, %Tesla.Env{status: 500}} -> get_numbers(page)
            _ -> handle_get()
        end
end

But now I get an error saying undefined function handle_get/0

6

Something like this:

def get_numbers(page) do
  "#{@base_url}#{page}"
  |> recursive_get()
  |> handle_get()
end

defp recursive_get(url) do
  case get(url) do
    {:ok, Tesla.Env{status: 500} -> recursive_get(url)
    anything_else -> anything_else
  end
end

Be careful that the recursive_get might DDOS the server, because there is no backoff strategy :slight_smile:

7

If I just want to check the status 500, how would I return the recursive_get function?

defp recursive_get(url) do
    case get(url) do
        {:ok, %Tesla.Env{status: 500}} -> recursive_get(url)
        if there is no status 500, send back to get_numbers() the request I got from get() and give it to handle_get()
    end
end
8

Well, I think my problem is somewhere else now… I did

defp recursive_get(url) do
    case get(url) do
        {:ok, %Tesla.Env{status: 500}} -> recursive_get(url)
        result -> result
    end
end

And put a IO.inspect() inside my get_numbers function

def get_numbers(page) do
        "#{@base_url}#{page}"
        |> recursive_get()
        |> IO.inspect()
        |> handle_get()
end

It returns me the body just fine until page 4 and then gives me an error saying there is no function clause matching in my controller

Here is my controller

def index(conn, _params) do
        Challenge.fetch_numbers()
        |> handle_response(conn)
    end

    defp handle_response({:ok, numbers}, conn) do
        conn
        |> put_status(:ok)
        |> json(numbers)
    end

    defp handle_response({:error, _reason} = error, _conn), do: error

I put a |> IO.inspect() after my Challenge.fetch_numbers() and it is returning me a nil, is it because of the result → result in the recursive_get()? How should I return if its a {:ok, %Tesla.Env{status: 200}}

9

The error message should give you some clues about where specifically something is going wrong; you haven’t posted the code of Challenge.fetch_numbers so it’s really hard to speculate about the cause.

10

Hey, this is the code inside my Challenge module

defmodule Challenge do
  alias Challenge.Number

  defdelegate fetch_numbers(), to: Number.Get, as: :call
end

The one inside the Number.Get module

defmodule Challenge.Number.Get do
    alias Challenge.ChallengeApi.Client
    alias Challenge.Number

    def call() do
        page = 1

        page
        |> recursive_call()
    end

    defp recursive_call(page) do
        page
        |> Client.get_numbers()
        |> handle_response()

        page = page + 1

        if page < 5 do
            recursive_call(page)
        end
    end

    defp handle_response({:ok, body}), do: {:ok, Number.build(body)}
    defp handle_response({:error, _reason} = error), do: error
end

And the one inside the Number module

defmodule Challenge.Number do
    @keys [:numbers]

    @derive Jason.Encoder
    defstruct @keys

    def build(%{"numbers" => numbers}) do
        %__MODULE__{
            numbers: numbers
        }
    end
end
11

if is a statement that always returns a value - if there’s no else and the condition is false, it returns nil:

iex(api@localhost)27> defmodule Foo do
...(api@localhost)27>   def bar(x) do
...(api@localhost)27>     if x do
...(api@localhost)27>       :ok
...(api@localhost)27>     end
...(api@localhost)27>   end
...(api@localhost)27> end
{:module, Foo,
 <<70, 79, 82, 49, 0, 0, 5, 136, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 0, 146,
   0, 0, 0, 17, 10, 69, 108, 105, 120, 105, 114, 46, 70, 111, 111, 8, 95, 95,
   105, 110, 102, 111, 95, 95, 10, 97, 116, ...>>, {:bar, 1}}

iex(api@localhost)28> Foo.bar(true)
:ok

iex(api@localhost)29> Foo.bar(false)
nil

Your code needs to store the value it gets from handle_response someplace.

12

How would I take this if off the code and stop all the recursion in case the next page returns me a empty json like {“numbers”: []}?

Like, if pages 1 to 5 have jsons with {“numbers”: [n1, n2, n3 … n]}, but page 6 and forth will always return me {“numbers”: []}, how could I stop the recursion and get all these numbers in a single one? I believe my build function already builds the single json that I need, but I can’t stop the recursion from happening without the if inside the recursive_call()

13

Could I do something like

defp recursive_call(page) do
        page
        |> Client.get_numbers()
        |> handle_response()

        page = page + 1

        cond recursive_call(page) do
            {:ok, body: %{"numbers" => [not empty]}} -> send to my first handle_response to Number.build
            {:ok, body: %{"numbers" => []}} -> stop the recursion and everything, dont put it inside my json and just return whatever it was inside the single json up to this point
        end
    end

Maybe it would be better to put this cond inside another private function and call it before handle_response?

14

While I was waiting for you answer to the other questions, I decided to check if the recursive requests are working and for some reason once I hit around page 300-400, the requests start again from page 1… Could it be a timeout on my side or maybe an overload on the external API server? I put a plug Tesla.Middleware.Timeout, timeout: 2_000 on the client module, should it be higher? Should I make the requests async? How do I do that in Elixir?

15

I found Task.async, but I don’t know how to use it correctly…

I put this in my code

defp recursive_call(page) do
        Task.async(fn ->
            page
            |> Client.get_numbers()
            |> handle_response()
        end)

        page = page + 1

        :timer.sleep(10)

        if page <= 10000 do
            recursive_call(page)
        end
    end

The page 10k is the last one with numbers inside the json, after that all pages give back an {“numbers”: []}, so I put it hard coded there so I could see where it would get and it went until around page 5000/6000 and started from page 1 again

16

Tesla has batteries included: Tesla.Middleware.Retry.