Fl4m3Ph03n1x
HTTP2 error using gun
Background
I have an application that uses gun to make HTTP requests to a server while keeping the connection up.
Problem
The problem here is that I am getting the following general stream errors:
2019-03-13 14:55:55.258 [error] GENERAL ERROR: {:stop, {:goaway, 0, :enhance_your_calm, "too_many_pings"}, :"Client is going away."}
2019-03-13 14:55:57.282 [error] GENERAL ERROR: {:stop, {:goaway, 5, :enhance_your_calm, "too_many_pings"}, :"Client is going away."}
2019-03-13 14:55:58.924 [error] GENERAL ERROR: {:stop, {:goaway, 3, :enhance_your_calm, "too_many_pings"}, :"Client is going away."}
GOAWAY
So, according to the HTTP2 spec:
The GOAWAY frame (type=0x7) is used to initiate shutdown of a connection or to signal serious error conditions. GOAWAY allows an endpoint to gracefully stop accepting new streams while still finishing processing of previously established streams. This enables administrative actions, like server maintenance.
So, I am guessing the server receiving my petitions isn’t too happy and is telling my client to slow down.
My confusion
This is confusing to me due to some reasons, the main one being that I am not attacking a single machine, I am attacking a cloud balancer that has a cluster behind. In theory, this balancer would distribute my load to all the servers and things like this wouldn’t happen.
I also don’t understand if gun simply closes a connection and opens again or if any data was lost.
Can someone help me understand the causes and consequences of this error?
Most Liked
peerreynders
Disclosure: Here I’m just as a clueless as the proverbial rubber duck.
But in the interest of making more information available:
- The error suggests to me that a PING frame is being answered with GOAWAY about every two seconds.
- Found a critical (possibly misinformed) opinion about the practice of HTTP/2 ping frames.
- The gun documentation suggests that the default HTTP/2 ping timeout is 5000ms.
ENHANCE_YOUR_CALM (0xb):
The endpoint detected that its peer is exhibiting a behavior that might be generating excessive load.
Do the errors change when you extend your configured timeout for gun? (Not expecting it to but it’s good to rule things out).
axelson
What cloud balancer is the service using? Perhaps the cloud balancer is doing some rate limiting?
shanesveller
Without knowing more about your circumstances, I can’t point to a concrete solution, but my hackles always go up when people speak in absolutes - and to call a problem “impossible” to solve on a major IAAS provider doesn’t sound likely.
For this specific concern, better use of connection draining, load-balancer health checks, and possibly some more graceful shutdown behavior of your own application, should be able to eliminate close to 100% of end-user 502s that stem from per-server interruptions like rolling deployments rather than true application defects.
This is part and parcel of zero-downtime deployments, which has been a desirable practice for quite some time, so we probably would’ve heard by now if it was truly unattainable on GCP. Their load-balancing is, in many ways, a bit more sophisticated than what AWS offers, for example.
Last Post!
Fl4m3Ph03n1x
We have had 502 error from Google since we can remember. We know our cluster has enough machines and we know they are not returning errors nor are they overloaded (we have metrics to check). We know these requests (502) never make it to our machines in the first place. I am not Sherlock Holmes, but after lengthy discussions about this topic we literally have no other explanation.
As for major IAAS providers, I worked for GCP for some time so I know for a fact (since I know how it used to work) this is totally possible. As a client I used Azure for a quite some time (definitely worst than GCP) and we had troubles every 2 weeks. It was a nightmare.
So to me, and with my experiences and knowledge I have, it sounds not just likely, but normal.
Google manages their own load balancers, which send traffic world wide to all the regions. We don’t have direct control over them (we have some limited control). This is the charm of the offer - you don’t need to worry too much with balancers, Google does.
It is worth mentioning that after a long stress test, we concluded these errors were around 0.0015% of all requests. This is well withing the terms of service Google providers iirc, so there is little that can be done. We are leaving this matter as 0.0015% is not a sufficient error rate to invest more time and effort.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"