thank you very much! , that’s solve my issue. I get no errors anymore logged.
Now I have to find out why the process still creating records although the end of the file was reached.
The file containing products:
cat file.csv | wc -l
10000000
Total products created in the database:
mysql> select count(*) from products;
+-----------+
| count(*) |
+-----------+
| 109958881 |
+-----------+
1 row in set (15.69 sec)
Can’t know but obviously empty the DB before starting a run.
That is what I do before each run. I also clear the log before to check that the amount of INSERT is equal to the amount of data records created into the database 
As said above, start chopping that CSV file on smaller chunks and experiment.
I can’t help you further unless you give me full access to the file, inform me of your exact MySQL version, what Erlang / Elixir versions you use, and give me access to the code.
You’ll have to track this down yourself because even if I was willing to help further and drill deeper, I am simply out of bandwidth to do so. That might change in just a few days or might not change for months – my work load varies.
I am pretty sure the problem is easy to solve though.
Oh btw, one final thing you can try just so we’re 100% sure:
Don’t do parallelism. Just stream the CSV file records and insert them one by one and see if you get 10M records.
BTW the amount above is mind-blowingly different… you expect 10M records but get almost 110M? Something is super weird, man, you must have overlooked something in the code.
I say make a public GitHub project with all your code and include a link to where the big CSV file can be downloaded (outside GitHub) so people can try and help you.
I have uploaded the file separately as advised. The file is available for download until 12.01.2024: products.csv
I have now created a repo with the code I use.
Thank you so much for your help!
What MySQL engine are you using? Is it MariaDB or good old MySQL version? And what version?
Dude, you have no database migration to create the table. Put one in! How can anyone help you if you haven’t took care to create the DB schema?
EDIT: I mean, it’s pretty simple, you should have included it:
defmodule Store.Migrations.AddProducts do
use Ecto.Migration
def change do
create table(:products) do
add(:name, :string)
add(:bar_code, :string)
add(:price, :string)
add(:currency, :string)
timestamps(type: :utc_datetime_usec)
end
end
end
You have a mistake in your data sample file. First line contains a shell prompt:
anonymized@ANONYMIZED:~/file_watch_example$ head -100
The rest is valid CSV.