Hi all,
I have an intermittent bug in production which I am trying to track down which causes requests to take a really long time:
Heroku times out all requests after 30 seconds, meaning the response never get’s sent for these slow requests and I never get alerted of the error.
Ideally I would force my own 30 second timeout to kill the request early. I’ve looked into cowboy’s idle_timeout option but it seems to have strange behaviour, not returning an error message to the user but rather a chrome error page and the client automatically attempts a refresh.
I am thinking of setting up a genserver that does a Process.send_after(__MODULE__, {:kill_if_open, request_pid}, 30_000)
and run Process.exit(pid, :kill)
in the :kill_if_open
handler.
Does anyone have any feedback on this approach? If it is bad practice, I could simply log/report that request and then investigate later.
I’ve now implemented it:
defmodule Myapp.RequestTimeout do
use GenServer
require Logger
@timeout :timer.seconds(2)
def start_link(_) do
GenServer.start_link(__MODULE__, %{}, name: __MODULE__)
end
@impl true
def init(_) do
{:ok, nil}
end
@impl true
def handle_cast({:register_request, log_meta, request_pid}, state) do
Process.send_after(__MODULE__, {:kill_if_open, log_meta, request_pid}, @timeout)
{:noreply, state}
end
@impl true
def handle_info({:kill_if_open, log_meta, request_pid}, state) do
Logger.metadata(log_meta)
Logger.error("RequestTimeoutError: #{Logger.metadata()[:request_id]}")
Process.exit(request_pid, :timeout)
{:noreply, state}
end
#
# API
#
def register_request(log_meta) do
GenServer.cast(__MODULE__, {:register_request, log_meta, self()})
end
end
defmodule Myapp.RequestTimeout.Plug do
def init(opts) do
opts
end
def call(conn, _) do
IO.inspect(Logger.metadata()[:request_id], label: "request_id")
Myapp.RequestTimeout.register_request(Logger.metadata())
conn
end
end
But there is no response sent to the user, the browser just hangs.
If anyone has any feedback or suggestions I’d be grateful to hear.
Thanks