Poor performance on EC2

[100phlecs]

This thread may be of interest to you.

Help with performance (file io)

One very relevant area with concurrency is when you need to process multiple files. We have to do something a lot like this at cargosense as part of parsing binary data files that get, and I’m able to do several hundred MB/sec of IO on my MBP by leveraging 4-8 files processing concurrently. With streams, this only uses about 10mb of memory.

I would try out File.stream! and doing more lines at once:

Getting fast I/O stream processing in Elixir does require some non-obvious tweaks. Due to the way I/O works on the BEAM (there’s a special process that does I/O and passes results to your process as a message), you want the messages to be as long as possible to avoid overheads.