Inserting CSV into postgres

sabri

Hello,

I have found lovely CSV library that I will be using to insert CSV files into postgres.

But there are some issues I am wondering about, as I am fresh in elixir.

Ok, here is the sample code I will be using to insert the CSV into DB:

File.stream!("ignore/customers.csv")
|> CSV.decode
|> Enum.each(fn
        {:ok, [id, nm, csr, sal]} ->
            Customer.changeset(%Customer{},
                %{masterid: id,
                  custname: nm,
                  csrid: String.to_integer(csr),
                  salesid: String.to_integer(sal)})
            |> Repo.insert
         {:error, message} ->
            # Whatever you want to do with invalid rows
    end)

My questions are:

In the Enum.each what is the time interval between each call to the Repo.insert? can I control this to make sure that my DB won’t get over-pumped with queries?
As I need to implement a progress bar in the browser while the CSV being inserted into DB, Can I broadcast a progress message in successful Enum.each to client using channels? for ex: after |> Repo.insert to broadcast message as:

MyApp.Endpoint.broadcast client_topic, “progress”, %{
progress: progress_number
}

15 comments

#ecto #streams

2 7147 15

2017-02-24 06:24:40 UTC

First Post!

Show Best Posts
Show All (oldest first)
Show All (newest first)

minhajuddin

If you have a lot of records and don’t need data validation use COPY to pipe the data to postgres. This is very fast.

The time between each insert is actually the time it takes to insert the record, So you insert a record and then another. So you probably don’t have to worry about overloading the server. Also, broadcasting a message should also be possible using the technique you mention

Post #1

Most Liked

hq1

I recently faced a similar task, so hopefully some of my experiences will come in handy.

COPY was my first thought. It’s the most efficient way to import CSV data to postgres, period.
It’s also the least flexible way from an Elixir app perspective. Building a custom query, handling separators, error handling/reporting, testing, finally “all or nothing” semantics (it’s a single transaction).

If you’re OK with the above, COPY is for you.

If you’re troubled with performance and still don’t want to completely sacrifice flexibility, consider using nimble_csv. Thanks to super clever implementation (metaprogramming and binary matching), sequential parsing is way faster than the library you’ve mentioned, that attempts parsing in parrallel (IIRC 5M rows in 20 seconds vs 2 minutes according to my microbenchmarks).

nibmble_csv works with Streams too, so you’ll be fine when it comes to memory spikes.

If you care about parallel processing later on, to make the DB insertion efficient (by utilizing the connection pool and bulk inserts), here’s what you can do to make it reasonably fast:

stream # can be a file, but you can use IO.stream for testing
|> YourNimbleParser.parse_stream(headers: discard_headers?) # nimble_csv discards it by default
|> Stream.chunk(1000, 1000, []) # experiment, fine tune
|> process_in_parallel(&chunk_handler_fn/1)

process_in_parallel implementation is entirely up to you. If you’re on Elixir 1.4, you may use async_stream; if lower than that, parallel_stream looks like an OK choice. Just make sure the number of parallel processes is somewhat in line with System.schedulers_online and your database pool size. Make it configurable, measure, rinse and repeat. How scheduling works.

Your chunk_handler_fn/1 will receive.. a chunk of 1000 decoded rows. You may prepare changesets there, have them validated, filter the chunk based on valid? property, remap the columns according to a custom mapping rule, build a list of maps to be inserted (changeset.changes is already there for you, perhaps needs to be enriched a little) and push it through Repo.insert_all in each individual process.

Caveats:

The parallel bulk inserts are not wrapped in a single transaction. You have to deal with individual chunk errors. Use on_conflict option if the need be to perform an upsert/ignore constraint errors.
You don’t really deal with individual rows. If you’d like to report progress you have to figure out the number of chunks upfront and emit a notification every time a batch was inserted/processed. A separate short-living process gathering the stats and forwarding progress could be a good idea.
It’s quite fast. In my case I was able to parse, validate/reject and insert 1M rows/minute on average. That was local, no latency penalty etc.
If you’re looking to report errors for individual rows and match them with CSV file line number, look up Stream.with_index. Note, that you will have to calculate the offset based on the header presence (the very first row gets either skipped or included).
nimble_csv will brutally crash on bad rows, e.g. discontinued quote. You might want to rescue from (catch) NimbleCSV.ParseError and convert it to something useful ({:error, {:parse_error, reason}} tuple most likely).

Hope this helps.

Cheers

edit added a note about NimbleCSV.ParseError handling, fixed grammer

Post #5

sabri

Thanks, I’ve finally implemented it!, but I have to admit, that I had to change my mindset to do it
Here is a sample chunk:

[%{first: “Antonio”, seq: “1”}, %{first: “Robert”, seq: “2”},
%{first: “Gordon”, seq: “3”}, %{first: “Tyler”, seq: “4”},
%{first: “Landon”, seq: “5”}, %{first: “Ethan”, seq: “6”},
%{first: “Amy”, seq: “7”}, %{first: “Paul”, seq: “8”},
%{first: “Tom”, seq: “9”}, %{first: “Helen”, seq: “10”}]

As I have used Stream.map in the pipeline:

File.stream!(path)
    |> MyParser.parse_stream
    |> Stream.map(fn n ->
      case n do
        [seq,first,last,email,digit] -> %{seq: seq, first: first}
        _ -> ""
      end
      end)
    |> Stream.chunk(10)
    |> Task.async_stream(&MyApp.CsvsController.chunk_handler_fn/1)
    |> Stream.run

Thanks all for support