Bulk inserts with Postgrex

[OvermindDL1]

@elixirnewbie

I am trying to insert 50,000 record

Wooo, quite a number, I’d probably use the PostgreSQL COPY command.

Some relevant threads:

Inserting CSV to Postgres via COPY

Hello, I have some troubles with importing of CSV data. Here is some snippet: s1 = "\"47\";\"-17756145\";\"some data\";\"type\";\"17756145\";\"50\";2017-07-27 13:34:28.095734;2017-07-27 13:34:28.095734" s2 = "\"47\";\"-17756146\";\"some data 2\";\"type\";\"17756145\";\"50\";2017-07-27 13:34:28.095734;2017-07-27 13:34:28.095734" stream = Ecto.Adapters.SQL.stream(MyRepo, "COPY areas(region_code,code,name,type,parent_code,addr_level,inserted_at,updated_at) FROM STDIN CSV DELIMITER ';' ") MyRep…

Inserting CSV into postgres

Hello, I have found lovely CSV library that I will be using to insert CSV files into postgres. But there are some issues I am wondering about, as I am fresh in elixir. Ok, here is the sample code I will be using to insert the CSV into DB: File.stream!("ignore/customers.csv") |> CSV.decode |> Enum.each(fn {:ok, [id, nm, csr, sal]} -> Customer.changeset(%Customer{}, %{masterid: id, custname: nm, csrid: String.to_integer(cs…

COPY FROM STDIN through Repo

I’m found example of COPY FROM STDIN in postgrex.ex on github: Postgrex.transaction(pid, fn(conn) -> query = Postgrex.prepare!(conn, "", "COPY posts FROM STDIN",[copy_data:true]) stream = Postgrex.stream(conn, query, []) Enum.into(File.stream!("posts"), stream) end) How I can convert these example to work with through Repo? Repo.transaction fn -> query = "some query" Ecto.Adapters.SQL.query!(Repo, query, []) #copy from stdin, how?? end