I think you’re receiving big amount of structures (in the
{:matcher, data}) structure
indeed; trying to see how the whole thing behaves with large amounts
Is hibernate an “ok” solution ? it kinda solved it all; but feels like a shortcut for things i dont understand (yet)
Running such genserver to receive loads of data from different inputs could just be a plain bad idea ?
hibernate is okay, but I don’t know why want to collect garbage straight away. I mean, gc will be called when there’s no space left, so you don’t have to worry about running out of memory (in this situation). And when garbage collection is initiated by VM, it will be more efficient in time, than calling GC manually every time
The gen_server process can go into hibernation (see erlang:hibernate/3) if a callback function specifies ‘hibernate’ instead of a time-out value. This can be useful if the server is expected to be idle for a long time. However, use this feature with care, as hibernation implies at least two garbage collections (when hibernating and shortly after waking up) and is not something you want to do between each call to a busy server.
What I would do is to return a short timeout value, but not zero, and manually call the garbage collector in handle_info/2.
playing around on heroku, on purpose as am learning this whole otp thing and would like to see where thing gets “heavy”
the thing is i dont see the no space left doing anything, rather system errors (memory quota exceeded). i might see whats up on other kind of instances
You can list all processes in your system, then for each get the memory usage, and then print that sorted in your console. Maybe other processes are consuming too much memory.
Your genserver is receiving large amount of data, yet does very little “work” on them, so gc was triggered late; I think gc is triggered by the amount of “work” since last gc. Just curious, why do you need a genserver here? You can just upsert the data in the original process; ecto will take care of the atomicity.
@lud running within phoenix so i opened the dashboard route to see whats up; might explain why below, it is not the memory of a given process, but the amount of genservers running
@derek-zhou what would you mean by ‘work’ ? you bring another issue with the upsert.
this genserver receives data from numerous other running ones. Doing so in the original processes (the numerous other ones) just timed out the db super fast. Centralizing the upsert within this single genserver solved the db timeouts. This surprised me a lot, as the amount of data and queries is the very same; it just flows another way.
But you’re right, i could totally do the subscribe/upsert in the main process, not in a genserver
My understanding of the GC is that it is triggered by every so many “reductions”, which is a BEAM defined metric of “work”. If running the upsert in all the originating processes jams the system, then serializing them on one process will cause serious backlog on its incoming message queue. You may have other problems.
Always super excited to start reading threads like these but almost all of them inevitably end with silence… 
so far no errors/missing data with the “working” 
approach - eg. dispatching through pubsub and centralizing the upsert in a single genserver; but am really wondering why such behaviour happens (the db timeout when each genserver was doing its upsert)