Cowboy handlers proably exit, not being logged or reported anywhere

What’s the log level set to in production? IIRC the “unexpected exit” reports are written at info.

Hmm, seems like a potentially similar issue to this recent (unresolved) post: Phoenix didn't handle some http requests

Most likely the request process crashed due to a link. Cowboy should report those, unless the exit reason is shutdown, which doesn’t log anything throughout OTP.

Given you know the request path, see if that path is starting or communicating with any process that might exit.

So what is happening is when a linked process crashes (as in throws an exception) I am getting crash reports properly:

  def index(conn, _params) do                                                                                                                                            
    Agent.start_link fn ->                                                                                                                                               
      raise "wat"                                                                                                                                                        
    end                                                                                                                                                                  
                                                                                                                                                                         
    render(conn, "index.html")                                                                                                                                           
  end 

09:31:10.288 [error] CRASH REPORT Process <0.1088.0> with 1 neighbours crashed with reason: #{'__exception__' => true,'__struct__' => 'Elixir.RuntimeError',message => <<"wat">>} in 'Elixir.UI.DashboardController':'-index/2-fun-0-'/0 line 8
09:31:10.289 [error] Cowboy stream 8 with ranch listener 'Elixir.UI.Endpoint.HTTP' and connection process <0.995.0> had its request process exit with reason: #{'__exception__' => true,'__struct__' => 'Elixir.RuntimeError',message => <<"wat">>} in 'Elixir.UI.DashboardController':'-index/2-fun-0-'/0 line 8

When a linked process exits, however, like this:

  def index(conn, _params) do                                                                                                                                            
    Agent.start_link fn ->                                                                                                                                               
      Process.exit(self(), :kill)                                                                                                                                        
    end                                                                                                                                                                  
                                                                                                                                                                         
    render(conn, "index.html")                                                                                                                                           
  end   

I only get the initla Plug.Logger line and then nothing, and the browser keeps spinning and loads this error:

errfailed1122×567 16.2 KB

Note: the exit reason can be anything, not just :normal, and Cowboy won’t report anything for me.

Similarly to zhangzhen 's problem, this is happening on prod when I handle webhooks (although from different system) and we did have issues that for example the JSON they were sending us didn’t comply with standard. I suspect there’s a similar issue here, that the machine-generated payload somehow is messed up and some linked process (or the process of handler) performs exit() ? rather than a crash.

My plan is, as I know the path that this is happening on, is to install a monitor from within a custom plug, and collect the exit reasons from requests on that path and maybe this will give me some idea what’s going on behind the scenes.

It’s a separate question if Cowboy / Plug / Phoenix should handle the situation more gracefully. In Cowboy’s documentation there are some hints that you want to monitor handler processes, like this one from here https://ninenines.eu/docs/en/cowboy/2.6/guide/handlers/:

This callback is optional because it is rarely necessary. Cleanup should be done in separate processes directly (by monitoring the handler process to detect when it exits).

but I understand that monitoring all handlers would have some performance punishment.

Can you try with something other than kill? That’s the “strongest” exit signal someone can submit and therefore we can’t generalize it.

Note this is not related to handlers though. Who is reporting this from Cowboy’s side is the connection, not the handler. And given Cowboy is already monitoring the handler process, I don’t see why it wouldn’t report other reasons too.

I have investigated this a bit, the issue is here:

https://github.com/ninenines/cowboy/blob/master/src/cowboy_stream_h.erl#L127-L141

Cowboy is not handling a {'EXIT', Pid, Whatever}, which is the format of exit reasons from linked processes. I would open up an issue on Cowboy and ask if they would consider adding a catch clause.

Oh, I did assume they don’t intend to handle it but looking at the code it is more likely an unintended bug. I will report to them.

Actually I looked at the code more in details in Cowboy and it looks like they have the try/catch clause and this is the only way they are catching these exits.

So I think it is meant to catch exits coming from the process itself (i.,e. using exit("SOMETHING") in Elixir), but if some other process sends it an exit signal (i.e. if using Process.exit(pid, "SOMETHING") thisi s not being caught unless process is trapping exits, but even then the :kill wouldn’t be captured I believe.

So it’s not as simple as adding clause here as it won’t be caught by try/catch clause. The only way to detect this situation I think is to monitor the process that is being sent an exit signal.

I will open an issue on Cowboy but I doubt this is something they want to handle. We’ll see.

This is not correct. cowboy_http is handling exits and the code I linked above is the result of receiving a {'EXIT', PID, Reason):

https://github.com/ninenines/cowboy/blob/master/src/cowboy_http.erl#L258-L260

Yes, verified that, you are correct. It only won’t catch :kill I think.