Writing data to a CSV file in chunks

Hello all :wave:,
I have a map like this -

%{
  "key1" => ["val1","val2","val3","val4","val5"]
  "key2" => ["val1","val2","val3","val4","val5"]
 }

now I need to write this into a csv like this -


I have researched about this and found that you can do this with NimbleCSV by-

  def write_csv_file() do
    data = key_mapping()
    File.write!("data.csv", [@bom, data_to_csv(data)])
  end

  def data_to_csv(data) do
    data
    |> Enum.map(fn {key, list} ->
      [key | list]
    end)
    |> MyParser.dump_to_iodata()
    |> :unicode.characters_to_binary(:utf8, {:utf16, :little})
  end

but currently, I am planning to do this in batches that means I will create batches of data and then write them to the CSV file, but I have no idea on how to do that without deleting the data written by the previous batch.
Any thoughts on how I should proceed?

Check docs for function File.write/3, the last argument has append mode.

1 Like