So, I am analyzing a crashdump from a crash I have with the slogan eheap_alloc: Cannot allocate 3280272216 bytes of memory (of type "heap").. I discovered the process that is trying to allocate that much memory, but when I try to open that process’s details on crashdump viewer it crashes with this exception:
05:52:23.005 [error] GenServer :crashdump_viewer_server terminating
** (ArgumentError) argument error
:erlang.tuple_size([:"#CDVIncompleteHeap"])
crashdump_viewer.erl:2768: :crashdump_viewer.parse_heap_term/4
crashdump_viewer.erl:1618: :crashdump_viewer.parse/3
crashdump_viewer.erl:2918: :crashdump_viewer.do_deref_ptr/4
crashdump_viewer.erl:2789: :crashdump_viewer.parse_tuple/6
crashdump_viewer.erl:1618: :crashdump_viewer.parse/3
crashdump_viewer.erl:2918: :crashdump_viewer.do_deref_ptr/4
crashdump_viewer.erl:2789: :crashdump_viewer.parse_tuple/6
Last message (from #PID<0.31427.4>): {:proc_details, '<0.27814.73>'}
State: {:state, '/Users/kelvinst/Downloads/erl_crash_core1.dump', [0, 5], 8, '54294'}
Client #PID<0.31427.4> is alive
(stdlib) gen.erl:169: :gen.do_call/4
(stdlib) gen_server.erl:219: :gen_server.call/3
cdv_proc_cb.erl:65: :cdv_proc_cb.get_details/2
cdv_detail_wx.erl:49: :cdv_detail_wx.init/1
wx_object.erl:372: :wx_object.init_it/6
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
So, I suppose the problem is happening because the actual heap of that process is too big to be saved, or something like that. So my next question would be: how to proceed with investigations on that issue on those cases? If the BEAM crashes because of a memory problem and the dump is not saved how will I discover which process did consume that much memory?
In my specific case, the dump had no information about the faulty process most of the times, but then we got lucky and one of the crashes did actually put it on the list. That was the dump that generated the error I mentioned on the previous post. In the end of the day I managed to find out the problem by looking at the readable stuff on the dump, like the =proc_stack of the culprit’s parent process, but that took me a long time to figure out, and a lot of tries going on the wrong direction.
As the BEAM is a really reliable runtime, I suppose there is actually another way to investigate that type of problem, or even that the dump problem we had is not common.
Have you seen the freely available online Erlang in Anger book? It’s about diagnosing these kinds of issues, the tools, etc…
1 Like
Hey! Nice one! I have red the “Memory Leaks” section and it already gave me ideas for the next time that happens, like using :erlang.system_monitor(). As in my case it was one process doing a very unoptimized query with lots of duplicated data, the system crashed for only that one process and it was kind of a nightmare to find out which process was that and what actually it was doing since the dumps were incomplete.
But still, is that something normal? I mean, getting these incomplete crash dumps? Or is it something I could configure to make sure that does not happen? I red the crash dump section on the book but didn’t found anything about that. I mean, even incomplete it helps a little by telling the total memory and that kind of stuff, now I know how to workaround that, I am just curious if that’s not something we could mitigate.
1 Like
When a system runs out of memory, a normal program or the BEAM, all reporting after that starts getting a bit… fuzzy… ^.^;
1 Like
True, the dump already helped a lot, even being incomplete could manage to discover the problem just by deep diving into it. I was just curious if it was just a specific case or that would always happen for memory issues.
That’s what happens when almost everything bug gets so easy to diagnose, you get used to that level of awesomeness. Thanks BEAM!
1 Like
