Fl4m3Ph03n1x
Background
I have an application that uses gun to make HTTP requests to a server while keeping the connection up.
Problem
The problem here is that I am getting the following general stream errors:
2019-03-13 14:55:55.258 [error] GENERAL ERROR: {:stop, {:goaway, 0, :enhance_your_calm, "too_many_pings"}, :"Client is going away."}
2019-03-13 14:55:57.282 [error] GENERAL ERROR: {:stop, {:goaway, 5, :enhance_your_calm, "too_many_pings"}, :"Client is going away."}
2019-03-13 14:55:58.924 [error] GENERAL ERROR: {:stop, {:goaway, 3, :enhance_your_calm, "too_many_pings"}, :"Client is going away."}
GOAWAY
So, according to the HTTP2 spec:
The GOAWAY frame (type=0x7) is used to initiate shutdown of a connection or to signal serious error conditions. GOAWAY allows an endpoint to gracefully stop accepting new streams while still finishing processing of previously established streams. This enables administrative actions, like server maintenance.
So, I am guessing the server receiving my petitions isn’t too happy and is telling my client to slow down.
My confusion
This is confusing to me due to some reasons, the main one being that I am not attacking a single machine, I am attacking a cloud balancer that has a cluster behind. In theory, this balancer would distribute my load to all the servers and things like this wouldn’t happen.
I also don’t understand if gun simply closes a connection and opens again or if any data was lost.
Can someone help me understand the causes and consequences of this error?
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Enabling companies to succeed by building software people love.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #ai
- #phoenix_html
- #elixirconf-us
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Most Liked- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
peerreynders
Disclosure: Here I’m just as a clueless as the proverbial rubber duck.
But in the interest of making more information available:
7. Error Codes
Do the errors change when you extend your configured timeout for
gun? (Not expecting it to but it’s good to rule things out).axelson
What cloud balancer is the service using? Perhaps the cloud balancer is doing some rate limiting?
shanesveller
Without knowing more about your circumstances, I can’t point to a concrete solution, but my hackles always go up when people speak in absolutes - and to call a problem “impossible” to solve on a major IAAS provider doesn’t sound likely.
For this specific concern, better use of connection draining, load-balancer health checks, and possibly some more graceful shutdown behavior of your own application, should be able to eliminate close to 100% of end-user 502s that stem from per-server interruptions like rolling deployments rather than true application defects.
This is part and parcel of zero-downtime deployments, which has been a desirable practice for quite some time, so we probably would’ve heard by now if it was truly unattainable on GCP. Their load-balancing is, in many ways, a bit more sophisticated than what AWS offers, for example.
Last Post!
Fl4m3Ph03n1x
We have had 502 error from Google since we can remember. We know our cluster has enough machines and we know they are not returning errors nor are they overloaded (we have metrics to check). We know these requests (502) never make it to our machines in the first place. I am not Sherlock Holmes, but after lengthy discussions about this topic we literally have no other explanation.
As for major IAAS providers, I worked for GCP for some time so I know for a fact (since I know how it used to work) this is totally possible. As a client I used Azure for a quite some time (definitely worst than GCP) and we had troubles every 2 weeks. It was a nightmare.
So to me, and with my experiences and knowledge I have, it sounds not just likely, but normal.
Google manages their own load balancers, which send traffic world wide to all the regions. We don’t have direct control over them (we have some limited control). This is the charm of the offer - you don’t need to worry too much with balancers, Google does.
It is worth mentioning that after a long stress test, we concluded these errors were around 0.0015% of all requests. This is well withing the terms of service Google providers iirc, so there is little that can be done. We are leaving this matter as 0.0015% is not a sufficient error rate to invest more time and effort.