Fl4m3Ph03n1x

Fl4m3Ph03n1x

Not getting response body with :gun

Background

Some time ago someone in this wonderful community suggested I used gun as an HTTP client, given that I was having severe issues with HTTPoison and later on had them with HTTPotion as well (they didn’t scale well enough).

Code

To fix it, we moved our solution to use the asynchronous HTTP client mentioned above: gun.
gun is an erlang library that communicates with a GenServer via events, calls and casts. To use gun I have therefore built a primitive GenServer client that prints to the console everything it receives:

defmodule ConnPoolWorker do
  use GenServer
  alias ProcessRegistry
  alias :gun, as: Gun

  @url_domain 'google.com'
  @https_port 443

  def start_link({worker_id}) do
    IO.puts("Starting worker #{worker_id}")

    GenServer.start_link(
      __MODULE__,
      nil,
      name: via_tuple(worker_id)
    )
  end

  ## Public API

  #makes a GET request via gun
  def fire(worker_id, url) do
    GenServer.cast(via_tuple(worker_id), {:fire, url})
  end
  
   ## Implementation 
   defp via_tuple(worker_id) do
    ProcessRegistry.via_tuple({__MODULE__, worker_id})
  end

  @impl GenServer
  def init(_args) do
    {:ok, conn_pid} = Gun.open(@url_domain, @https_port)
    {:ok, _protocol} = Gun.await_up(conn_pid)
    {:ok, conn_pid}
  end

  @impl GenServer
  def handle_cast({:fire, url}, conn_pid) do
    Gun.get(conn_pid, url)
    {:noreply, conn_pid}
  end

  # handle_info everything else
  @impl GenServer
  def handle_info(msg, state) do
    IO.puts("MSG: #{inspect msg}")
    {:noreply, state}
  end
end

This client is registered in the Registry using via_tuples, but I don’t think that is majorly important for now.

Problem

This code works. It opens a connections, waits for the connection to be up, and if you invoke fire to make a request (get it? because the library is called gun? :smiley: ) you get a :gun_response event that handle_info picks up.

However, that’s precisely the issue. It’s the only event the process ever picks up. This process gets no other events like :gun_data or :gun_trailers, even though the documentation says it should.

The only thing I get every now and then is a :gun_down (connection down) followed by a :gun_up (connection up) which is normal:

MSG: {:gun_response, #PID<0.214.0>, #Reference<0.1474087065.2991849473.15656>, :fin, 302, [{"server", "nginx"}, {"date", "Wed, 20 Feb 2019 11:55:14 GMT"}, {"content-length", "0"}, {"connection", "keep-alive"}, {"location", "http://www.sapo.pt/noticias/"}, {"strict-transport-security", "max-age=31536000"}, {"x-content-type-options", "nosniff"}, {"content-security-policy", "upgrade-insecure-requests; block-all-mixed-content"}, {"x-xss-protection", "1; mode=block"}, {"referrer-policy", "origin-when-cross-origin"}]}
MSG: {:gun_down, #PID<0.221.0>, :http, :closed, [], []}
MSG: {:gun_down, #PID<0.224.0>, :http, :closed, [], []}

Question

  1. Am I doing something wrong with my GenServer client?
  2. Is this working as intended or am I missing something?
  3. How can I get the rest of the data from the request?

Marked As Solved

Fl4m3Ph03n1x

Fl4m3Ph03n1x

I found the issue. Turns out I was getting the full response:

{:gun_response, #PID<0.214.0>, #Reference<0.1474087065.2991849473.15656>, :fin, ....}

The last argument :fin means this is the only response I am supposed to get. So, both :gun and my server are working as intended as I am getting a redirection code 302.

Also Liked

amnu3387

amnu3387

I think in your case continue could work fine because you’re instantiating independent genservers that then receive messages to do their work, although doing the bootstrap on init is probably better in this particular case (I say this not knowing the remaining of the system).

Still, to understand how {:continue ...} works you need to look into the process message box queue erlang uses.

Say you have GenServer “Server” and now, concurrently from any part of your system you have multiple processes sending “Server” a message

p1 ! Server :message_a
p2 ! Server :message_b
p3 ! Server :message_c

Now the BEAM guarantees that the messages are delivered in order to the process “Server” (caveats apply if processes are spread on nodes and not on a single one)

So “Server” has in its mailbox: [:message_a, :message_b, :message_c]

And it handles one by one, handle_info(any, state), so the first handle matching will be called with msg a, then b, then c.

If for instance you have a handle that continues, say for :message_b

handle_info(:message_b, state), do: {:noreply, state, {:continue, :recalc}}

What happens is, back to the same 3 sent messages

[:message_a, :message_b, :message_c]
Processed A
[:message_b, :message_c]
Processed B
:continue
[:continue_recalc, :message_c]
Processed handle_continue :recalc
[:message_c]
Processed C
[]

So it basically hijacks the message box and puts on its first item whatever is the continuation handle, and that is guaranteed to run before any other message is processed. That continuation handle can itself continue to something else, and if it did, that would be guaranteed to be processed as well before any other message.

In your case, let’s say you start ConnPoolWorker.start_link({1}) on your supervision tree

def init(_) do
   {:ok, :not_ready, {:continue, :open}}
end

Here it returns ok, the supervisor receives the :ok, and continues starting the remaining tree, somewhere else on your app down the line, some other process are started and they start sending fire requests, say 5000 , the message box now has 5000 requests, but you’re guaranteed that before trying to process any of those, the continue handle will be executed.

def handle_continue(:open, _) do
   {:ok, conn_pid} = Gun.open(@url_domain, @https_port)
   {:ok, _protocol} = Gun.await_up(conn_pid)
   {:noreply, conn_pid}
end

So only when this handle continue executes are the remaining messages processed.

Of course if this handle_continue fails (say Gun.open errors out), then the genserver will crash, and those messages already sent and waiting on the process’ message box will be lost, whereas if you start it as you do in init, and there’s no way for the processes down the line to be started and start firing requests unless the init is successful, it might be better to simply do the bootstrap on init (as you’re doing) and prevent the app from even going further if the gun connections can’t be open.

But if you don’t loose the “targets” until the requests are successfully done, meaning if the app/processes crashes when they restart you’ll be able to fire the same requests because they haven’t been completed, there wouldn’t be a problem in using :continue.

One example where you probably don’t want to use :continue is when your genserver instantiates things that won’t be interacted by its own message box (like say, you’re populating a big ETS table that is bootstrapped from a DB) and this ETS is going to be read directly by other processes. In this case using :continue to populate the ETS table could mean that other processes start trying to access the ETS before it’s fully populated/even created, while populating it on init would guarantee that it’s ready before the Supervision Tree continues starting the other bits of your system that might interact with it. But even in this case, it might be a reasonable tradeoff, it depends.

shanesveller

shanesveller

I would strongly encourage moving the connection building out of your init into a handle_continue if you will be spooling off a lot of these in rapid succession, which feels likely for an HTTP client wrapper. init is blocking until it returns, which means anything calling start_link on this module is blocking too.

Fl4m3Ph03n1x

Fl4m3Ph03n1x

An excellent response, that I have bookmarked for future reference !

Where Next?

Popular in Questions Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

Other popular topics Top

malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
Nvim
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New

We're in Beta

About us Mission Statement