aura999

aura999

Issue with streaming into a csv

I am trying to stream into a csv file using nimble_csv(1.2.0) but running into issues.

This is the code I am using to create a csv file from the stream.

defmodule Utils
  alias NimbleCSV.RFC4180, as: CSV

  def write_to_file(content_stream) do
    path = Path.join(System.tmp_dir(), "#{UUID.uuid4(:hex)}.csv")

    content_stream
    |> CSV.to_line_stream()
    |> CSV.parse_stream([skip_headers: false])
    |> Stream.into(File.stream!(path, [:write, :utf8]))
    |> Stream.run()
  end
end

This is my test data

contents = [
    "First Name,Last Name,Email\n",
    "David,Byrne,david@test.com"
  ]
  |> Stream.map(&String.trim_leading/1)
   
Utils.write_to_file(contents)

The file is created however the contents in the file are missing the comma separator
and the new line characters

First NameLast NameEmailDavidByrnedavid@test.com

If however I update Utils.write_to_file/1 to by removing parse_stream/2

def write_to_file(content_stream) do
    path = Path.join(System.tmp_dir(), "#{UUID.uuid4(:hex)}.csv")

    content_stream
    |> CSV.to_line_stream()
    |> Stream.into(File.stream!(path, [:write, :utf8]))
    |> Stream.run()
end

I get the csv contents in the correct format.

First Name,Last Name,Email
David,Byrne,david@test.com

I am not sure if I am missing any options that I need to pass to parse_stream/2 that would cause the contents to be malformed.

Any help would be appreciated

Marked As Solved

LostKobrakai

LostKobrakai

Streams are lazy enumerables. Being a stream doesn’t tell you anything about what data the streams deals with. parse_stream maps from a stream of csv formatted binaries to a stream of rows being lists of cell contents. It’s the lazy version to parse_enumerable, which immediately returns [[binary()]].

In your case I’m really confused though because if you already have a csv file and you want to write that csv file, then there’s no need for NimbleCSV in the first place.

Enum.into(content_stream, file_collectable) would be all you need.

What makes streams even more confusing here is that a File.Stream struct implements both Enumerable.t (a - in this case lazy – collection to enumerate) as well as Collectable.t (a collection, which items can be pushed into), which work independently. You’re using the latter functionality here.

Last Post!

aura999

aura999

Thanks for the detailed info @LostKobrakai. Appreciate it :+1:

Where Next?

Popular in Questions Top

lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

Other popular topics Top

joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement