I’ve done further testing and noticed some performance and memory usage gains by flattening IOData more efficiently.

The benchmarking results show a large difference between the methods, which I’m skeptical about. However, it does seem to point in a promising direction. I ran into issues when trying to read straight from the file during the benchmarking, so I had to store the data in a variable before running the tests. The document I tested was a standard 5.9MB MS Word file.

Here’s the function I created:

defmodule StreamIodata do  
  def flatten_iodata_stream(iodata_stream, opts \\ []) do
    chunk_size = Keyword.get(opts, :chunk_size, 65_536)

    Stream.transform(
      iodata_stream,
      fn -> [] end,
      fn iodata, acc ->
        acc = [acc | iodata]
        acc_size = IO.iodata_length(acc)

        if acc_size >= chunk_size do
          # Flatten the accumulated iodata into a binary
          binary = IO.iodata_to_binary(acc)
          # Reset the accumulator
          { [binary], [] }
        else
          # Keep accumulating
          { [], acc }
        end
      end,
      fn
        # After function to flush any remaining data
        acc when acc == [] -> []
        acc -> [IO.iodata_to_binary(acc)]
      end
    )
  end
end

And here’s the benchmark:

defmodule Bench do
  def benchmark_docx_parsing(data) when is_list(data) do
    Benchee.run(
      %{
        "Stream with flatten_iodata_stream (8kb)" => fn ->
          data
          |> StreamIodata.flatten_iodata_stream(chuck_size: 8_192)
          |> Saxy.parse_stream(TestDocxHandler, [])
        end,
        "Stream with flatten_iodata_stream (64kb)" => fn ->
          data
          |> StreamIodata.flatten_iodata_stream(chuck_size: 65_536)
          |> Saxy.parse_stream(TestDocxHandler, [])
        end,
        "Stream with flatten_iodata_stream (524Kb)" => fn ->
          data
          |> StreamIodata.flatten_iodata_stream(chuck_size: 524_288)
          |> Saxy.parse_stream(TestDocxHandler, [])
        end,
        "Stream with iodata_to_binary map" => fn ->
          data
          |> Stream.map(&IO.iodata_to_binary/1)
          |> Saxy.parse_stream(TestDocxHandler, [])
        end
      },
      time: 10,
      memory_time: 2
    )
  end
end

# Storing the file in memory to avoid issues when reading from the file during benchmarking
zip_file = 
  path
  |> Unzip.LocalFile.open()

{:ok, unzip} = Unzip.new(zip_file)

data =  
  unzip
  |> Unzip.file_stream!("word/document.xml")
  |> Enum.into([])

Bench.benchmark_docx_parsing(data)

Here’s the output:

Error trying to determine erlang version enoent, falling back to overall OTP version
Operating System: macOS
CPU Information: Apple M1
Number of Available Cores: 8
Available memory: 8 GB
Elixir 1.17.2
Erlang 27
JIT enabled: true

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
reduction time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 56 s

Benchmarking Stream with flatten_iodata_stream (524Kb) ...
Benchmarking Stream with flatten_iodata_stream (64kb) ...
Benchmarking Stream with flatten_iodata_stream (8kb) ...
Benchmarking Stream with iodata_to_binary map ...
Calculating statistics...
Formatting results...

Name                                                ips        average  deviation         median         99th %
Stream with flatten_iodata_stream (524Kb)      872.32 K        1.15 μs  ±4444.97%        0.88 μs           4 μs
Stream with flatten_iodata_stream (64kb)       871.69 K        1.15 μs  ±4267.88%        0.88 μs        3.04 μs
Stream with flatten_iodata_stream (8kb)        753.12 K        1.33 μs  ±6291.89%        0.88 μs        4.33 μs
Stream with iodata_to_binary map                 1.75 K      570.29 μs    ±68.51%      507.23 μs     1230.03 μs

Comparison: 
Stream with flatten_iodata_stream (524Kb)      872.32 K
Stream with flatten_iodata_stream (64kb)       871.69 K - 1.00x slower +0.00083 μs
Stream with flatten_iodata_stream (8kb)        753.12 K - 1.16x slower +0.181 μs
Stream with iodata_to_binary map                 1.75 K - 497.47x slower +569.14 μs

Memory usage statistics:

Name                                         Memory usage
Stream with flatten_iodata_stream (524Kb)         1.77 KB
Stream with flatten_iodata_stream (64kb)          1.77 KB - 1.00x memory usage +0 KB
Stream with flatten_iodata_stream (8kb)           1.77 KB - 1.00x memory usage +0 KB
Stream with iodata_to_binary map                896.40 KB - 505.46x memory usage +894.63 KB

**All measurements for memory usage were the same**