Storing list of structs in file one by one


I’m having a list of elixir structs and I need to store in into file for later use and I don’t want to serialise it.
So I used :erlang.term_to_binary on the list and that worked fine, but sometimes it happens that there is too much data and system kills the process because it’s out of memory.

So I was thinking about converting it one by one using streams, I have something like this

|> item -> :erlang.term_to_binary(item) end)
|> Stream.into(!(dir <> "/data.dat"))

However I’m not sure if this is actually somehow better then just simply :erlang.term_to_binary(values_list), but I assume it could be better as it’s one by one, I’m I wrong?

Another problem is that I don’t know how to actually read the file properly. I guess I should add some binary mark, where one item ends, right?

Sorry for probably novice question, I’m still getting to know elixir.

Thank you in advance.

Maybe use dets instead? Or mnesia if you need something more sophisticated.

1 Like

Thanks, this looks promising.

1 Like