In that case the only likely way to reduce the footprint is to try to clear out whatever is processed before you move to the next step. I’ve never done it before but I’ve thought about it a lot because I expected it might be a challenge with the copy on write approach.
Potentially send out pieces you’re working with to their own process so that the smaller chunks can easily be garbage collected?
Just speculating.
One thing to look out for if you’re using ets tables to store binary fragments retrieved from the xml, you might be causing your source xmls to never get gc’d, since a fragment is just a offset+length on the original binary
I find myself in violent agreement with NobbZ statement “trusting those external inputs to not spill my atom table, monitored or not, is out of the question.” I also wonder how many Elixir libraries (to say nothing of app-specific modules) are trusting external inputs not to crash the BEAM.
One of the most pleasant things about Elixir is that we don’t have to worry about all of the pitfalls introduced by thread programming. However, it strikes me that there may be a set of Elixir-specific pitfalls. Is there a “best practices” document that lays out potentially unsafe coding practices of this sort? If not, perhaps some of the smarter folks on the list could create one…
-r
I needed to parse large (1-3gb) XML documents with hundreds of thousands of records recently. My first attempt was using SweetXML and XPath but the memory consumption was very high, so I switched to ultimately switched to Saxy successfully. It was a bit confusing to get started, dealing with pretty complex XML files, but my solution ended up working well.
I’d be happy to go into greater detail if anyone’s interested, but just wanted to second using Saxy.
SweetXML was great for a smaller set of data, it just got unwieldy.
Just to call out you can use them in tandem - have your Saxy handler spit out xmerl and then you can use SweetXML to query into that.
Just to follow up with what we ended doing at work after a lot of research.
We now use the default Saxy handler to spit out simple form, then we use data_schema’s to query the simple form. We implemented our own querying for simple form - which was not to difficult to do in the end. (If you are interested I may be able to open source).
This means you can cast the strings in the XML into data values quickly, then traverse the results using Elixir functions rather than xpath.
Not using xmerl and SweetXML drastically reduced our memory impact (we are talking from 1gb to 80mb for large XMLs) and sped it all up by about 12 times.
On top of that I’ve also implemented a custom Saxy handler that will only create the intermediate simple_form representation for values that are required by a data schema. So you pass a schema to the saxy handler then Saxy only creates simple_form for the values that the schema wants.
This keeps the memory impact very low as it never spikes higher than what is required by the schema. It adds a little complexity though.
I ran into this issue of atom exhaustion vulnerability recently. In my case, this would have been a major issue because I was looking to parse SAML assertions received via a publicly accessible endpoint.
I ended up going a similar route as @Adzz and others by using saxy and its SimpleForm output.
To help avoid this repeated pain for others, I’ve created the following library, which avoids this atom exhaustion problem and includes the ability to verify XML signatures.
I welcome any community contributions or feedback 
.