dfens
Memory leak with quantum and processing binary files
Currently I have phoenix app which also schedules quantum jobs (GitHub - quantum-elixir/quantum-core: ⌚ Cron-like job scheduler for Elixir · GitHub) with overlap: false mode
Jobs are importing & exporting relatively large json files.
Now my problem is
Leaking memory
This is current status and it will get worse since different/bigger data sets are going to be processed.
For example on start on the left app starts with ~60MB
Flat line in the end is after quantum job was processed
I did check also this https://stackoverflow.com/questions/43613027/solving-large-binaries-leak but forced gc / tuning ERL_FULLSWEEP_AFTER collect did not seem to help
If I run ImportTask/ExportTask on my local machine but not as quantum and observe memory - then I cannot reproduce this behaviour
Any tips welcome 
Marked As Solved
NobbZ
Also Liked
dfens
@NobbZ jackpot!
as the matter of fact I was using Jaxon for parsing since files were pretty big before. but just tested with Jason strings: :copy and issue does not occur anymore. I suspect this was misuse of Jaxon.stream on my side but I am completely satisfied with Jason results
@benwilson512 yes I did try, without any effect
Thank you so much for your input guys
NobbZ
From a quick glance it seems as if jaxon wasn’t copying and also did not provide an option to copy binaries.
You might still be able to use jaxon when you :binary.copy/1 strings it gives you manually.
Last Post!
dfens
I suspected enum map but it does only this:
Insert chunk into postgres (let’s say ~1000 items with subresources)
returns struct %ChunkRepory{success: 700, failed: 300}
map is aggregated later to sum of those maps,
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"