Binary string to struct causing massive memory overhead?

Questions / Help
1

Hi, I’m new to Elixir and trying to understand what might be happening here:

I have structured text data that I’m attempting to store as nested structs. From what I’ve read, the fields of a struct should be stored only once in memory and then the data using those structs references the shared struct definition.

Using Process.info/1 and :hibernate to see and minimize memory in an actor storing this data, I’m seeing my 2k binary strings become up to 80k when converted to nested structs. Additional strings consume a similar amount of memory when converted.

Any ideas on what I might be doing terribly incorrectly? Is this actually expected behavior? Any suggestions would be appreciated, thanks!

2

Hey @darkmarmot, are you able to provide any example of the data or code you’re using? It’s a bit hard to offer suggestions or advice with the problem this abstract.

3

Quoting the tutorial, if you use the update syntax to transform a struct:

iex> john = %User{}
%User{age: 27, name: "John"}
iex> meg = %{john | name: "Meg"}
%User{age: 27, name: "Meg"}

Then John and Meg will share keys. You can exploit this when building a large list of structs.

It’s not necessarily shared between all User instances though, and messaging or sending to ETS likely removes the sharing. I’d be curious to check if it applies to two structs created “from scratch”…

4

@benwilson512 I’ll need to make a test application to show the issue outside of my current work.

Another person pointed me to this, which I think points to the same problem – with some potential improvements coming in OTP 21.

Thanks!

2 Likes