Memory leak with quantum and processing binary files

Have you tried strings: :copy (assuming you use Jason)?

Have you tried calling :erlang.garbage_collect() at the end of whatever function you’re having Quantum run?

@NobbZ jackpot!
as the matter of fact I was using Jaxon for parsing since files were pretty big before. but just tested with Jason strings: :copy and issue does not occur anymore. I suspect this was misuse of Jaxon.stream on my side but I am completely satisfied with Jason results

@benwilson512 yes I did try, without any effect

Thank you so much for your input guys

From a quick glance it seems as if jaxon wasn’t copying and also did not provide an option to copy binaries.

You might still be able to use jaxon when you :binary.copy/1 strings it gives you manually.

Basically I have let’s say 100MB file

What I was doing was like:

 path
           |> File.stream!()
           |> Jaxon.Stream.query([:root, :all])
           |> Stream.chunk_every(@chunk_size)
            |> Enum.map(... process chunk)

now when process did finish there was like XXXMB leftover per each iteration. Anyway what I am searching for is kind of stream parser and it seems that code below is misuse because memory spikes immediately to ~1.5GB and drops sequentialy with each chunk (not as stated with jaxon)

That is probably because of Enum.map, depending on what it does exactly, it will force the full stream.

Also if you keep references to binaries in that Enum.map instead of using :binary.copy/1 as suggested, you might keep all the lines in memory until the sub references get collected.

But overall, this is hard to say without knowing how you process the chunks…

I suspected enum map but it does only this:

Insert chunk into postgres (let’s say ~1000 items with subresources)
returns struct %ChunkRepory{success: 700, failed: 300}

map is aggregated later to sum of those maps,