Use Enum.uniq_by

Enumerates the enumerable, by removing the elements for which function fun
returned duplicate items.

The function fun maps every element to a term. Two elements are considered
duplicates if the return value of fun is equal for both of them.

The first occurrence of each element is kept.

## Example

    iex> Enum.uniq_by([{1, :x}, {2, :y}, {1, :z}], fn {x, _} -> x end)
    [{1, :x}, {2, :y}]
    
    iex> Enum.uniq_by([a: {:tea, 2}, b: {:tea, 2}, c: {:coffee, 1}], fn {_, y} -> y end)
    [a: {:tea, 2}, c: {:coffee, 1}]

How large is the CSV? Enum.uniq_by won’t help you detect duplicates, it’ll just remove them.

If the goal is to have something like “no products should have duplicate names” or similar then the easiest route is to just define a uniq database constraint. Then load all the csv rows in a single transaction, and rollback the transaction if one of the inserts fails.

Ah that’s not too bad at all. Just load it into memory, look for duplicates, error if there are some, and if there aren’t use Repo.insert_all.

If you end up with 10s of thousands to 100s of thousands of rows or more you can look at other options.