Add an artifical row into a stream

Questions / Help
, ,
1

I have a table in Postgres holding file uploads. I also have a requirement to download a subset of these files as a zip file to the user.

In order to keep a handle on memory and disk IO, I have managed to stream this content straight from the DB to the client. What I have is code that uses Ecto.stream to pluck 10 rows at a time from the DB, send the relevant data to Zap (a streaming zipfile builder), which is then chunked out to the client. All very neat and it works.

def send_zip_contents(id, chunker) do
  query = << an ecto query returning file metadata and contents >>

  one_mb = 1024 * 1024

  Repo.transaction fn ->
    query
    |> Repo.stream(max_rows: 10)
    |> Stream.map(fn upload -> {upload.name, upload.contents} end) 
    |> Zap.into_stream(one_mb)
    |> chunker.()
  end
end

Unfortunately, I need to add one extra file to that zip archive. A summary file that’s not in the DB. My question is this: Is there a way to add one more Zap Entry tuple, i.e. {somefile.name, somefile.contents}, into that stream without breaking it? Some way, for instance, of having Stream.map produce one extra element?

2

@petelacey https://github.com/elixir-lang/elixir/blob/v1.10.2/lib/elixir/lib/stream.ex#L1103 Stream.concat/2 is your friend here:

   Repo.transaction fn ->
    query
    |> Repo.stream(max_rows: 10)
    |> Stream.map(fn upload -> {upload.name, upload.contents} end)
    |> Stream.concat([{extra_file.name, extra_file.contents}])
    |> Zap.into_stream(one_mb)
    |> chunker.()
  end
3 Likes
3

So simple. I swear, I went over the Stream API half-a-dozen times before posting, and this never clicked. Thank you!