How to split string into multiple chunks by size

Use binary pattern matching together with list comprehension:

for <<chunk::size(chunk_size)-binary <- input>>, do: chunk

This could produce invalid Unicode strings though?

@hauleth I did a quick test and it seems like this method will throughway a leftover chunk.

for <<chunk::size(5)-binary ← “hello world”>>, do: chunk
[“hello”, " worl"]

I found an answer from StackOverflow which goes like this:

 "hello world" 
|> String.codepoints
|> Enum.chunk_every(5)
|> Enum.map(&Enum.join/1)
# ["hello", " worl", "d"]

However, I not sure is there any performance implication here that I should take into consideration.

well, you could use Stream.chunk_every, so it would lazily chunk your string instead of eagerly. For long strings like you said you are working with, it should improve performance a lot already.

One question though, where are you getting this string from?

The stackoverflow is likely to be quite a bit slower.

handling utf8 is slow, but if not needed:

Here is a version based on the list comprehension but which takes the leftover into account/

defmodule Chunker do

  alias Chunker

  def chunk(string, size \\ 5), do: chunk(string, size, [])

  defp chunk(<<>>, size, acc), do: Enum.reverse(acc)
  defp chunk(string, size, acc) when byte_size(string) > size do
    <<c::size(size)-binary, rest::binary>> = string
    chunk(rest, size, [c | acc])
  end
  defp chunk(leftover, size, acc) do
    chunk(<<>>, size, [leftover | acc])
  end


  def stackoverflow(string, size \\ 5) do
    string
    |> String.codepoints
    |> Enum.chunk_every(size)
    |> Enum.map(&Enum.join/1)
  end

  def withstream(string, size \\ 5) do
    string
    |> String.codepoints
    |> Stream.chunk_every(size)
    |> Enum.map(&Enum.join/1)
  end

end

And here is the benchee run:

{:ok, data} = File.read("./data/alice29.txt")
Benchee.run(
  %{
    "chunk" => fn -> Chunker.chunk(data, 5000) end,
    "overflow" => fn -> Chunker.stackoverflow(data, 5000) end
  })

Here is the result:

Operating System: Linux
CPU Information: Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz
Number of Available Cores: 8
Available memory: 15.55 GB
Elixir 1.10.4
Erlang 22.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 21 s

Benchmarking chunk...
Benchmarking overflow...
Benchmarking withstream...

Name                 ips        average  deviation         median         99th %
chunk          586726.39     0.00170 ms  ±1225.64%     0.00145 ms     0.00350 ms
withstream         17.50       57.15 ms    ±14.33%       54.27 ms       81.87 ms
overflow           15.71       63.64 ms    ±16.49%       61.00 ms       94.93 ms

Comparison:
chunk          586726.39
withstream         17.50 - 33528.78x slower +57.14 ms
overflow           15.71 - 37339.36x slower +63.64 ms

Hi @kelvinst I tried to port a method createFileFromText from azure-storage npm to my ex_azure_storage lib. So to be honest I don’t know where the string comes from

Random thought here but if how UTF-8 uses a variable number of bytes bothers you or causes issues, you could use UTF-32 which is fixed-length!

Alternatively you might have to implement something which is smart enough to know how many bytes each grapheme requires. You might want to check out elixir/lib/elixir/unicode/unicode.ex at v1.11.4 · elixir-lang/elixir · GitHub

Basically read a chunk of bytes and consume graphemes repeatedly off it (and emit chunks whenever you have one that is large enough), store the rest in the buffer and continue reading, repeat until done

@cmkarlsson impressive I’ve learnt something today thanks to you

I see, yeah, I was just curious cause normally big amounts of data like that come from files or uploads, that could be streamed themselves, avoiding to load the whole thing to memory. But in your case, as it’s a ported lib function that takes a loaded string, you don’t much control over that, so yeah, binary pattern matching seems like the way to go then.