Inconsistent behaviour with Stripe webhooks. Not passing Plug.Parsers randomly

Hey,

I am experiencing something strange with Stripe webhooks.

We have a setup based on this article: https://connerfritz.com/blog/stripe-webhooks-in-phoenix-with-elixir-pattern-matching

We noticed some failing webhooks on the Stripe dashboard that say they got no response body (which should not happen, we send a response on every case from the relevant controller).

I started to look into it, and it looks like the request does not make it to the controller at all.

I added a plug to log when a step is passed in the endpoint.ex file.

  plug xxxWeb.StripeValidationPlug

  plug xxxWeb.LoggerPlug, msg: "passed stripe validation plug"

  plug Plug.Parsers,
    parsers: [:urlencoded, :multipart, :json],
    pass: ["*/*"],
    json_decoder: Phoenix.json_library(),
    # Max 1GB
    length: 1_000_000_000

  plug xxxWeb.LoggerPlug, msg: "passed Parsers plug"

Based on these logs it looks like we don’t make it past Plug.Parsers.

There are no errors, or any logs that would explain why this happens.

The only thing we see is that the request comes in and there is no response.

On other requests we see that the request came and response went:

Dec 16 09:57:11 AM  02:57:11.239 request_id=FzEmpgYZGousJoYADq7B [info] GET /healthz
Dec 16 09:57:11 AM  02:57:11.247 request_id=FzEmpgYZGousJoYADq7B [info] Sent 200 in 7ms

But for this we only see the incoming part:

Dec 16 09:57:07 AM  02:57:07.705 request_id=FzEmpTNucr4YJUoADq2B [info] POST /stripe_callback

I tried the same Stripe event, and sometimes it passes sometimes it doesn’t.

Any idea why this might be?
Or ideas to debug this?
Only happens on prod, seems rock solid locally/staging.

Thanks