gheorghina
Expired ssm-agent session where a debug node is connected via "iex --remsh" does not disconnect automatically the node, leaving nodes() state inconsistent; bonus: CPU on service node spikes instantly to 100%
Hi everyone,
The title of this issue is long, because the problem is a bit complicated.
I would need your support in telling me what I can investigate further in the BEAM.
I have a system of Tasks deployed in ECS(AWS), independent nodes for one service.
We are using AWS SSM(Working with SSM Agent - AWS Systems Manager) to connect to one of the desired nodes ( example: aws ssm start-session --target ${CLUSTER}${FARGATE_TASK_ID}${FARGATE_CONTAINER_ID})
Next step from the node is to connect to the elixir console for the service via iex --remsh service-name --sname debug --cookie COOKIE
We identified that if the ssm session expires, the debug node remains connected, and throws the CPU of the service to 100%.
In order to save the situation a manual/code disconnect_node action is being required, making the CPU drop back.
My questions are:
- how can I investigate what makes the CPU grow?
- How can I differentiate via a monitoring solution this case to automatically disconnect the node?
Thank you
Most Liked
gheorghina
Thank you for your input, unfortunately due to regulations in my case, we don’t connect to the exact service node, we connect to a debug node via which we have the chance to run
iex --remsh service-name --sname debug --cookie COOKIE
I did a bit more research to try to identify at the tcp level the hanging connections with the idea that maybe I can identify what is causing CPU increase by checking the Process statuses of the hanging PIDs, but I was not very lucky.
The node stats as well don’t show much in terms of increased reductions or run_queues. Which makes me clueless.
I will close the topic on my side for now. Thank you again.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"