Be sure to use String.graphemes/1 if there is any possibility of Unicode data in your strings, codepoints can do potentially not what you want. See here for more: String — Elixir v1.20.2
John-Goff
2
csokun
Thank @John-Goff I do want to support Unicode string.
Finally, with all your help I managed to put together a working code (decided to go with stream option for now) https://github.com/csokun/ex_azure_storage/blob/fileshare_create_file/lib/azure_storage/azure_file_share.ex#L177-L189
1
hauleth
What if I would upload file that content is not valid UTF-8? In general in this case you should not care at all about encoding of the file and use @cmkarlsson solution, but even then you do not need to construct list.
0
axelson
Scenic Core Team
FYI that snippet will discard the last partial chunk if the string does not divide by chunk_size
iex(1)> for <<chunk::size(3)-binary <- "abcdefgh">> do chunk end
["abc", "def"]
(if the last partial chunk was included then it would also return "gh")
Here’s a simple recursive function that will return the last chunk (and not return any empty strings)
@doc """
Chunks a string into chunk of a given byte size (NOT unicode safe)
## Examples
iex> chunk_string("abcdefgh", 3)
["abc", "def", "gh"]
iex> chunk_string("abcdef", 3)
["abc", "def"]
"""
def chunk_string(string, chunk_size) when chunk_size > 0 do
case string do
<<chunk::size(chunk_size)-binary, rest::binary>> ->
[chunk | chunk_string(rest, chunk_size)]
"" ->
[]
rest ->
[rest]
end
end
1
bdarla
A relevant library that came to my attention recently is text_chunker_ex by Revelry (thanks to @hugobarauna for sharing this in Elixir Radar 420).
The aim of the library is to split text to be fed to AI models. As I understood, it mimics some functionality of the LangChain. I have not used it (yet), but seems interesting!
5
jpc-ae
This is a bit of an old one, but here’s my version that isn’t as memory-friendly as streaming, but it is unicode-safe and better than splitting+joining graphemes. Overall its quite performant:
Regex.split(~r/(.{5000}[\p{Mn}\p{Me}]*|.+)/su, string, include_captures: true, trim: true)
It’ll return a list of strings chunked into, in this case, 5000 characters (plus any combining marks to ensure they don’t get split up from what they’re attached to), with the remaining bit at the end. This won’t be exactly 5kb depending on how many multi-byte characters you have, but should do fairly well for most purposes. You can always adjust the match string to better reflect the input data structure.
0