vivus-ignis
Emulate File.stream! for a string variable
Hello there!
I have a mix task which grabs some data from a remote API, collects it into a file (a biggish xml), then sucks it in as a stream and processes doing a bunch of text transformations.
Now I’m trying to sketch an integration test for the processing part (skipping the getting-the-data part) and I wonder… is it possible to use a string variable instead of a file for that purpose?
This is what I have in my mix task code:
File.stream!("#{@download_dir}/#{category}.xml", [:read])
|> ... processing part I want to test follows
And this is how I’m trying to simulate (unsuccessfully so far) the File.stream! part:
input_xml = """
... xml fragment ...
...
"""
input_xml
|> Stream.unfold( &(String.split(&1, "~n")) )
|> ... processing part ...
Of course what I’m getting after this Stream.unfold is different from File.stream! – the Stream.unfold results in all the newlines being removed.
And then my processing part breaks as it relies on newlines in certain places (yeah, it sounds crazy, but inside that xml I have wiki markup-formatted fragments where newlines do matter).
So my question is: is it possible to split a string by newlines in such a way that I can preserve those "\n"s? That is, can I emulate FIle.stream! without an actual file?
Would appreciate any hints. Thank you!
Marked As Solved
wmnnd
You’re in luck, you can simply use StringIO.open/1 and IO.binstream in order to stream any String:
{:ok, stream} =
"abc\ndef\nghi\n"
|> StringIO.open()
stream
|> IO.binstream(:line)
|> #your own stream processing
Also Liked
ku1ik
Alternative solution to using StringIO + IO.binstream would be to use Stream.unfold with binary pattern matching:
def binary_stream(b, chunk_size \\ 50_000) when is_binary(b) do
Stream.unfold(0, fn skip ->
case b do
<<_skipped::binary-size(skip), chunk::binary-size(chunk_size), _rest::binary>> ->
{chunk, skip + chunk_size}
<<_skipped::binary-size(skip)>> ->
nil
<<_skipped::binary-size(skip), chunk::binary>> ->
{chunk, skip + byte_size(chunk)}
end
end)
end
I tested this on pretty big XML docs (hundreds of megabytes) and it seems to be performant and doesn’t require much memory due to not copying/cloning any part of the binary data.
PS: for this particular case (XML parsing), it’s not necessary to read by line, and in fact some XML documents (or SOAP API responses) return whole XML doc as a single long line without line breaks.
michalmuskala
Why not use one of the string functions? I understand you want to get an enumerable of lines from the string. This can be achieved eagerly with String.split(str, "\n") or lazily with String.splitter(str, "\n").
NobbZ
Because String.split/2 and String.splitter/2 will remove the split-points, but the OP said he needs them intact.
Last Post!
antoine
Thanks for this post, it was very usefull !
But I think the post marked as solution is not the more appropriate.
The solution using String.splitter(str, "\n") seems to behave more as expected.
Explanations:
When using this:
str = "abc\ndef\nghi\n"
{:ok, stream} = str |> StringIO.open()
s = stream |> IO.binstream(:line)
it do not work as expected:
iex> s |> Enum.take(1)
["abc\n"]
iex> s |> Enum.take(1)
["def\n"]
=> The result should always be the same as it’s the same operation.
Like we have here:
iex> s = File.stream!("/tmp/foo.csv")
iex> s |> Enum.take(1)
["hey\n"]
iex> s |> Enum.take(1)
["hey\n"]
Instead, String.splitter(str, "\n"), do the job as expected:
iex> str = "abc\ndef\nghi\n"
iex> s = String.splitter(str, "\n")
iex> s |> Enum.take(1)
["abc"]
iex> s |> Enum.take(1)
["abc"]
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Practical resources that improve the lives of professional developers.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"