Blackjaguar

Blackjaguar

How to send chunks of a zipped set of CSV files using a Stream

Good afternoon!

I am working on a project where users have the possibility to export data from different Psql tables as CSV.
Inspired by a great article (https://joaquimadraz.com/csv-export-with-postgres-ecto-and-phoenix) I managed to successfully setup a Stream to query and send the csv data as chunks to the user, where they are made available as a download.

This works great for one table at a time, but today I attempted to expand this as such that the user is able to request for multiple tables and receive the data in csv format zipped (compressed) into one file. However, I cannot get my head around how to achieve this.
Zipping data in a lazy manner seems counterintuitive to me, is this possible in Elixir/Erlang? It appears to me I need a structure that lazily reads rows from a Psql copy command, while zipping and also keep track of where one file stops and another starts.

How can I compress the data and send it as one package using a Ecto.Adapters.SQL stream and send it chunked to the user?

I hope I phrased the question clear enough, please let me know if more explanation is needed.

Marked As Solved

dimitarvp

dimitarvp

Off the top of my head:

defp produce_report(query, converter_function, header, file_path)
when is_function(converter_function) do
  {:ok, _} = Repo.transaction(fn() ->
    # This function assumes the result will be written to a file, not returned as a value.
    query
    |> Repo.stream
    |> converter_function.(header, file_path)
    |> Stream.run
  end, timeout: :infinity)
end

defp produce_report_users() do
  produce_report(query_users(), &YourReportModule.convert_user_to_csv_and_append_to_file/2, users_header(), users_file_path())
end

defp produce_report_achievements() do
  produce_report(query_achievements(), &YourReportModule.convert_achievement_to_csv_and_append_to_file/2, achievements_header(), achievements_file_path())
end

def produce_reports_users_and_achievements() do
  # Start all tasks asynchronously in the background.
  users_task = Task.async(&produce_report_users/0)
  achievements_task = Task.async(&produce_report_achievements/0)
  # Wait until all tasks are done.
  results = Task.yield_many([users_task, achievements_task], :infinity)
  #...<your code verifying the list of tuples that is the "results" variable goes here>...
  #... 
  # Create the resulting ZIP file. 
  zip_file = :zip.create(result_path(), [users_file_path(), achievements_file_path()])
  # ...<your code to handle the created ZIP file goes here>...
end

For more options on processing the results of many asynchronous tasks (as used in this very crude example), please see Task.yield_many docs.

Also check Erlang’s zip module.

Again, this is very rough but it should give you a general idea. I opted to spawn each report in a separate process which should be beneficial on multi-core machines; however, if each report is quite heavy and needs a lot of CPU, then you might want to just iterate over each report in-place (without spawning processes) – that depends on your priorities.

Also Liked

dimitarvp

dimitarvp

It would be great if you give a step-by-step example, giving several table names, input parameters and expected output – because I can’t quite follow.

I’ve done streaming and in-place CSV reports in Elixir before and it can be confusing when you first do it but rest assured, it’s very doable.

dimitarvp

dimitarvp

FYI you can mark mine as the accepted answer and it’s considered closed.

Glad I could help! Hope this made you understand Elixir better. :slight_smile:

Where Next?

Popular in Questions Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New

Other popular topics Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36432 110
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement