Questions & Help>Questions

Streaming XML to file

#xml
bettio

bettio

Streaming XML to file

Hello :slight_smile:

We are trying to export a “huge” amount of data from a database using XML.
Our XML exporter service is running on a cluster and it has a limited amount of memory (e.g. 4GiB of RAM) so we cannot keep in memory the whole XML document.
Does anyone knows any good Elixir (or erlang) library which allows to stream data so we don’t have to keep in memory the whole document before writing it?

15 comments
#xml
2960 15

First Post!

Most Liked

dimitarvp

dimitarvp

I hate to bring this suffering upon you but for excellent performance you should definitely just use Erlang’s :xmerl. It’s not exactly easy to work with, ask away if you get stuck.

It does support streaming, both reading and writing, so RAM will not be a problem.

4
Post #3
ananthakumaran

ananthakumaran

We had a similar use case the difference being we need to generate xlsx files (which is just a zip file with multiple xml files). Apparently neither google nor hex.pm seems to show this library for xml stream
https://github.com/activesphere/xml_stream

2
Post #10
bettio

bettio

This does not help:

Fast Expat based Erlang XML parsing library. I’m looking for a writer, not a parser.

1
Post #2

Last Post!

bettio

bettio

It works fairly well for us :slight_smile:

0
Post #15

Where Next?

View thread on forum (has 15 replies!)
xml
Home Questions & Help>Questions
#xml
8 2960 15
Last post
REPLY VIEW THREAD WITH 15 POSTS

Popular in Questions Top

nobody
Questions & Help>Questions

How to get the server ip address?
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
/phoenix
19 49590 16
New
jononomo
Questions & Help>Questions

Where / How does the Mix environment variable get set?
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
#mix
31 23259 9
New
PeterCarter
Questions & Help>Questions

Good auth solutions for Elixir/Phoenix?
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
/phoenix#authentication
48 21684 20
New
stefanchrobot
Questions & Help>Questions

How to decode a JSON into a struct safely?
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
#structs#json
29 21432 26
New
aadeshere1
Questions & Help>Questions

Write while loop equivalent in elixir
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
#while#loop
60 29182 36
New
aalberti333
Questions & Help>Questions

Enum.map over list of key/value pairs with a map as the value
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
#enummap
7 20041 6
New
WestKeys
Questions & Help>Questions

Mint vs Finch vs Gun vs Tesla vs HTTPoison etc
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
#http_client
253 22785 30
New

Other popular topics Top

electic
Questions & Help>Questions

DateTime Format to string?
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
#datetime
19 25125 16
New
minhajuddin
Questions & Help>Questions

Using List.first instead of Enum.at(0)
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
#code-style
76 33965 14
New
aadeshere1
Questions & Help>Questions

Write while loop equivalent in elixir
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
#while#loop
60 29182 36
New
dogweather
Chat & Discussions>Discussions

The complexity of Haskell vs. Elixir's simplicity
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
#language-implementation#haskell
138 30279 35
New
siddhant3030
Questions & Help>Questions

How can I write a raw sql query?
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
/phoenix#ecto
13 19888 20
New
SoCreat
Questions & Help>Questions

What's the best ide/editor for elixir in 2021?
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
#ide
68 38000 26
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

EEF

Supporting innovation across the BEAM ecosystem.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Error tracking for Elixir devs who love to ship. Start your free account.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Error tracking for Elixir devs who love to ship. Start your free account.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Software Mansion

The team behind Membrane, Popcorn, LiveDebugger. Available for hire.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement

Options

Show First Posts