How do I cast and insert many items of the same type at once?

Questions / Help
1

The items come from external source and I believe they should be cast or validated first. Should I trust Repo.insert_all() ?

Repo.insert_all( Item, [ [name: “first-item”], [name: “second-item”]])

2

external source

Should I trust Repo.insert_all() ?

Probably not.

You can run each of these items through validations, and if all are ok, use Repo.insert_all.

items
|> Stream.map(fn item_attrs -> Item.changeset(%Item{}, item_attrs))
|> Enum.split_with(fn %Ecto.Changeset{} = item_changeset -> item_changeset.valid? end)
|> case do
  {valid_item_changesets, []} ->
    raw_items_data =
      valid_item_changesets
      |> Stream.map(fn %Ecto.Changeset{} = item_changeset -> Ecto.Changeset.apply_changes(item_changeset) end) # turns item changesets into a list of %Item{}
      |> Enum.map(fn %Item{} = item -> # turns %Item{} into a map with only non-nil item values (no association or __meta__ structs)
        item
        |> Map.from_struct()
        |> Stream.reject(fn # or something similar
          {_key, nil} ->
            true
          
          {key, %_struct{}} ->
            # rejects __meta__: #Ecto.Schema.Metadata<:built, "items">
            # and association: #Ecto.Association.NotLoaded<association :association is not loaded>
            true 
          
          _other ->
           false
        end)
        |> Enum.into(%{}) # not really necessary since `insert_all` also accepts a list of lists.
      end)
      # maybe filter for empty maps/lists
    
    Repo.insert_all(Item, raw_items_data)

  {valid_item_changesets, invalid_item_changesets} ->
    # can insert valid and return a partially erred result (with errors from invalid_item_changesets)
    # or inserts neither and be "atomic"
    # depends on your use case
end

Although you would probably want to separate the above into several functions.

5 Likes
3

Many thanks for the response. That is what I was doing right now after you advised to run each item through validation check. But I was not so fast to finish those functions.

I’m still wondering if its the only way to do this task, I think its a common task that many have.

4

You can create a multi with all your items also, but it will be less efficient than using Repo.insert_all.

alias Ecto.Multi

def mega_item_multi(multi, []), do: multi
def mega_item_multi(multi, [{item, transient_item_index} | items]) do
  multi
  |> Multi.insert(transient_item_index, Item.changeset(item))
  |> mega_item_multi(items)
end

# need some "transient" index to later lookup items in multi map (%{1 => %Item{}, 2 => %Item{}, ...})
items_with_index = Enum.with_index(items)

# and call it like this
Multi.new()
|> mega_item_multi(items_with_index)
|> Repo.transaction()
|> case do
  {:ok, %{1 => %Item{}, 2 => %Item{}, ...}} -> # all items inserted successfully
  {:error, failed_transient_item_index, probably_some_invalid_item_changeset, successfully_inserted_items} -> # ...
end

Probably a bad idea.

1 Like
6

Why so rude?