Here’s a graph showing the memory reported by AWS. You can see when we switched back to Bandit around 8:30pm.
For our use case (predominantly WS messages served over http behind an AWS ALB inside AWS ECS), it appears we’re seeing 1/2 the memory usage with cowboy.
We look forward to returning to Bandit when the memory is more comparable!
ok, I’m going to have to ask a really dumb question because I don’t know what tool you’re using.
Is the memory utilization:
If bandit is increasing the buff/cache memory, then it is not a real memory increase. This is some weird artifact of hibernate or something. There is probably a tool to find out the actual used memory, but I’ve forgotten what it is.
$ free
total used free shared buff/cache available
Mem: 16162188 7218888 4205420 1177868 6264756 8943300
Swap: 16777212 260952 16516260
I compared the memory based on the processes in BEAM via live_dashboard and they looked comparable between cowboy and bandit. Perhaps it’s due to the server being idle, etc. not doing the thing that increases the memory, etc.
A couple of things that would greatly help:
config MyAppWeb,
http: [
...
websocket_options: [compress: false]
]
{:ok, pid} = Bandit.PhoenixAdapter.bandit_pid(YourAppWeb.Endpoint)
{:ok, connection_pids} = ThousandIsland.connection_pids(pid)
# You may need to turn the following a few times until you get one back that has `handler_module: Bandit.WebSocket.Handler` listed in its state
:sys.get_state(Enum.random(connection_pids))
This will be a raw dump of everything your socket process has in state; feel free to redact as needed (you’ll probably want to elide the whole elem(1).connection.websock_state; that’s all of your Phoenix state and should be identical between Bandit and Cowboy). Also, if you want to DM me here to further limit any exposure, that’s fine too.
Looking forward to hunting this down!
At Supabase we have experienced :long_schedule warnings of over 30s at our production load for our logging service that would then result in a subsequent crash. I have not yet had the time to open up an issue in the Bandit repo yet and put together a case for it, but the changes that made the warnings go away was the cowboy revert here, which makes me feel that Bandit still has some way to go in terms of performance at scale…
Unsure if my particular case is related to the memory issues others have also experienced as per this thread, but just putting it out there in case anyone else has a similar experience.
I’d suspect it’s the same issue that @jjcarstens et al saw last week, and which is fixed in Thousand Island 1.3.3+. If you’re able, give Bandit a spin with Thousand Island bumped appropriately and see if it fixes your issue; I’d love to get as many data points as possible for perf issues like this!
I think we’ve got a fix.
Anyone able to try the branch referenced here, I’d love to see more evidence that this resolves the issue. The main wrinkle in this solution is that GCing every request is using a sledgehammer on a nail; I’m hoping that I’ll be able to pull the mitigation back to something more reasonable
@scottmessinger and others; 1.3.0 just went out with a fix for the long-standing issue of increasing memory use (which wasn’t really a memory issue so much as it was an issue with how memory use is reported, but anyway), along with a few other fixes. See CHANGELOG as always. Hopefully this fixes your issue; please report back if so as I’d love to have more evidence that we’ve finally licked this.
Yup – I had meant cowboy.