Error insert many_to_many

hristonev · February 17, 2018, 11:35am

Hi, guys
I’m trying to insert many to many relation. Example is from José Valim - Wats new in Ecto 2.0 book.
Here is my implementation (copy):

schema "resources" do
  field :url,           :string
  field :status_code,   :integer
  field :hash,          :string
  field :size,          :integer
  field :headers,       {:map, :string}
  field :response_time, :integer
  belongs_to :mirror, Mirror
  many_to_many :anchors, Anchor, join_through: "resources_anchor"
  timestamps()
end

  def changeset(struct, params \\ %{}) do
    struct
    |> Ecto.Changeset.put_assoc(:anchors, insert_and_get_all(params.anchors))
  end

  defp insert_and_get_all([]), do: []
  defp insert_and_get_all(anchors) do
    hashes = Enum.map(anchors, &(&1[:hash]))
    Repo.insert_all(Anchor, anchors, on_conflict: :nothing)
    Repo.all(from a in Anchor, where: a.hash in ^hashes)
  end

When I try to insert record via changeset I got error:

** (FunctionClauseError) no function clause matching in Ecto.Changeset.put_assoc/4

anchors: #Ecto.Association.NotLoaded<association :anchors is not loaded>

No record inserted in “resources”. Records inserted only in “anchor”. When I remove put_assoc record is inserted in “resources” as well.

dokuzbir · February 17, 2018, 3:43pm

Make sure first argument is a changeset not a map in Ecto.Changeset.put_assoc(struct, :anchors, insert_and_get_all(params.anchors))

hristonev · February 17, 2018, 3:59pm

This is my call.

changeset = Resource.changeset(%Resource{
  url: url,
  status_code: status_code,
  hash: get_hash(body),
  size: byte_size(body),
  headers: headers |> Enum.into(%{}),
  mirror: mirror,
  response_time: response_time
}, %{anchors: parse_anchor(body)})

Next is insert

Repo.insert(changeset)

I’m not sure what you mean 1st argument changeset.

dokuzbir · February 17, 2018, 4:10pm

You cant use a struct directly in put_assoc. You need a changeset.

struct
|>Ecto.Changeset.cast(params, allowed_params) #add that function before put_assoc
|> Ecto.Changeset.put_assoc(:anchors, insert_and_get_all(params.anchors))

idi527 · February 17, 2018, 6:37pm

Does the example from the book also calls inserts inside changeset/2?

I would probably use Ecto.Multi for that. It would be something like this

alias Ecto.Multi

@spec create_resource(map) :: {:ok, %{anchors: [%Anchor{}], resource: %Resource{}} | {:error, :anchors | :resource, error :: term, changes_so_far :: map}
def create_resource(resource_attrs) do
  Multi.new()
  |> insert_and_get_anchors_multi(resource_attrs.anchors)
  |> insert_resource_multi(resource_attrs)
end

defp insert_and_get_anchors_multi(multi, anchors) do
  # seems a bit wasteful to me
  # probably would change it as well ...
  Multi.run(multi, fn _changes_so_far ->
    hashes = Enum.map(anchors, :anchors, &(&1[:hash]))
    Repo.insert_all(Anchor, anchors, on_conflict: :nothing)
    Repo.all(from a in Anchor, where: a.hash in ^hashes)
  end)
end

defp insert_resource_multi(multi, attrs) do
  Multi.run(multi, :resource, fn %{anchors: anchors}) ->
    attrs
    |> Resource.changeset() # changeset doesn't call put_assoc
    |> Ecto.Changeset.put_assoc(:anchors, anchors)
    |> Repo.insert()
  end)
end

As for your approach, the problem is spelled out in the error

** (FunctionClauseError) no function clause matching in Ecto.Changeset.put_assoc/4

anchors: #Ecto.Association.NotLoaded<association :anchors is not loaded>

put_assoc/4 expects something other than #Ecto.Association.NotLoaded<association :anchors is not loaded> from insert_and_get_all(params.anchors).

Maybe inspect what you get from insert_and_get_all(params.anchors) when your operation fails.

hristonev · February 17, 2018, 8:00pm

I’ll try your method @idi527. Believe I understand it better than changesets.

hristonev · February 17, 2018, 8:22pm

My last post is stupid :101: I’ll need to learn more about changesets and Ecto. It is very different from Doctrine. And its hard for me.

dokuzbir · February 17, 2018, 8:48pm

Sorry i didnt explain clearly. What i mean is %Ecto.Changeset{} struct… Cast function takes allowed params in your params, converts your %Resource{} strtuct to a %Ecto.Changeset{} struct. With that way put_assoc function matches. But as i see book has different example.

hristonev · February 17, 2018, 9:03pm

Guys, it’s alive . Thanks so much for your help. My mistake was that I do not declare map type, when calling changeset. It was %{} anon map, needed to be %Resource{} map. Now I’ll try Multi, this will do it in 1 query :). For PHP/Symfony guy may to be hard but not impossible.

hristonev · February 19, 2018, 12:07pm

@idi527 I’m trying to do this piece by Multi.

def changeset(struct, params \\ %{}) do
  struct
  |> Ecto.Changeset.cast(params, @allowed_params)
end

My changeset for non related params.

def create(struct, params \\ %{}) do
  anchors = insert_and_get_all(params.anchors)
  Multi.new()
  |> Multi.run(:resources, fn anchors ->
    Resource.changeset(struct, params)
    |> Ecto.Changeset.put_assoc(:mirror, params.mirror)
    |> Ecto.Changeset.put_assoc(:anchors, anchors)
    |> Repo.insert()
  end)
  |> Repo.transaction()
end

Create function like your “create_resource”
There is error in Muli:

** (FunctionClauseError) no function clause matching in Ecto.Repo.Schema.insert/4

If I use this code without Multi everything work.

idi527 · February 19, 2018, 3:47pm

I’d put insert_and_get_all(params.anchors) inside multi, otherwise it’s pointless, I think.

Note that you reassign anchors inside |> Multi.run(:resources, fn anchors -> so anchors = %{} there and you call put_assoc/3 with an empty map, which probably causes the error.

Resource.changeset(struct, params)
|> Ecto.Changeset.put_assoc(:mirror, params.mirror)
|> Ecto.Changeset.put_assoc(:anchors, %{})
|> Repo.insert()

But to be of more use, I’d need to see the full error and, if possible, stack trace as well.

hristonev · February 19, 2018, 5:49pm

How to get stack trace? In multi just top error is shown.

idi527 · February 19, 2018, 5:52pm

Stack traces are usually generated by default for most elixir exceptions like FunctionClauseError. And look like this

** (FunctionClauseError) no function clause matching in Ecto.Repo.Schema.insert/4
     stacktrace:
       (<appname>) <filename>:<linenumber>: <module>.<function>/<arity>
       (<appname>) <filename>:<linenumber>: <module>.<function>/<arity>
       (<appname>) <filename>:<linenumber>: <module>.<function>/<arity>
       ...

hristonev · February 19, 2018, 6:26pm

This is all info for this exception:

[error] Process #PID<0.477.0> raised an exception
** (FunctionClauseError) no function clause matching in Ecto.Repo.Schema.insert/4
(ecto) lib/ecto/repo/schema.ex:157: Ecto.Repo.Schema.insert(Frezu.Repo, Ecto.Adapters.Postgres, {:ok, %{resources: %Frezu.Resource{__meta__: #Ecto.Schema.Metadata<:loaded, "resources">, anchors: #Ecto.Association.NotLoaded<association :anchors is not loaded>, hash: "f625ce161ed86b19c30cad5502060e992168dcce130f869d45297b1ffb5f4e59", headers: %{"Cache-Control" => "no-store, no-cache, must-revalidate", "Content-Type" => "text/html;charset=utf-8", "Date" => "Mon, 19 Feb 2018 10:10:58 GMT", "Expires" => "Thu, 19 Nov 1981 08:52:00 GMT", "Pragma" => "no-cache", "Server" => "Apache", "Set-Cookie" => "PHPSESSID=m2jbcng6a43994b2hjgsftjsj7; path=/", "Transfer-Encoding" => "chunked"}, id: 99509, inserted_at: ~N[2018-02-19 10:11:00.650930], mirror: %Frezu.Mirror{__meta__: #Ecto.Schema.Metadata<:loaded, "mirrors">, html: false, id: 255, image: false, inserted_at: ~N[2018-02-19 10:10:48.152604], resource: false, site: %Frezu.Site{__meta__: #Ecto.Schema.Metadata<:loaded, "sites">, delay: 10, id: 2, inserted_at: ~N[2018-01-02 09:15:24.460979], updated_at: ~N[2018-02-05 08:53:04.647488], url: "http://nestinarka.net", user: #Ecto.Association.NotLoaded<association :user is not loaded>, user_agent: "Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0", user_id: 18}, site_id: 2, updated_at: ~N[2018-02-19 10:10:48.155136]}, mirror_id: 255, response_time: 1840895, size: 24151, status_code: 200, updated_at: ~N[2018-02-19 10:11:00.650939], url: "http://nestinarka.net/bg/gallery/rooms/"}}}, [])
(frezu) lib/frezu/Crawler/parser.ex:19: Frezu.Parser.parse/4

idi527 · February 19, 2018, 6:43pm

Ecto.Repo.Schema.insert/4 accepts a struct as the third argument, and in your case a tuple {:ok, ...} is passed.

Try replacing Multi.run with Multi.insert and

I’d put insert_and_get_all(params.anchors) inside multi, otherwise it’s pointless, I think.

hristonev · February 20, 2018, 7:18am

I have build something that work with Multi.

def changeset(struct, params \\ %{}) do
  struct
  |> Ecto.Changeset.cast(params, @allowed_params)
end

def create(struct, params \\ %{}) do
  Multi.new()
  |> Multi.run(:anchors, &insert_and_get_anchors(&1, params.anchors))
  |> Multi.run(:resources, &insert_and_get_resorce(&1, struct, params))
  |> Repo.transaction()
end

defp insert_and_get_resorce(%{anchors: anchors}, struct, params) do
  Frezu.Resource.changeset(struct, params)
  |> Ecto.Changeset.put_assoc(:mirror, params.mirror)
  |> Ecto.Changeset.put_assoc(:anchors, anchors)
  |> Repo.insert()
end

defp insert_and_get_anchors(_, []), do: []
defp insert_and_get_anchors(_, anchors) do
  hashes = Enum.map(anchors, &(&1[:hash]))
  IO.inspect(anchors)
  Repo.insert_all(Anchor, anchors, on_conflict: :nothing)
  {:ok, Repo.all(from a in Anchor, where: a.hash in ^hashes)}
end

All DB request now are in multi. Debug shows me

Insert into table anchors (1 query)
Select from anchors (1 query)
Insert into resources (1 query)
Many inserts into resources_anchor (many queries)

Seems that this queries run independent. Above each one I have

[debug] QUERY OK db=0.4ms or similar exec time

All data now is saved to DB but I still have an error at the end of “create”

[error] Process #PID<0.457.0> raised an exception
** (FunctionClauseError) no function clause matching in Ecto.Repo.Schema.insert/4                                                                                                  
(ecto) lib/ecto/repo/schema.ex:157: Ecto.Repo.Schema.insert(Frezu.Repo, Ecto.Adapters.Postgres, {:ok, %{anchors: [%Frezu.Anchor{__meta__: #Ecto.Schema.Metadata<:loaded, "anchors">, hash: "f35eff6ccf5fcf9825a2e7c5d89c25d47e4268786262c7602360adc50a3af84a", href: "http://nestinarka.net/bg/", html: "<a href=\"http://nestinarka.net/bg/\" title=\"Нестинарка - къщи за гости\"><img src=\"images/logo.png\" alt=\"Нестинарка - къщи за гости\" style=\"margin: 0 0 0 0;\"/></a>", id: 308699, rel: "n/a", target: "n/a", title: "Нестинарка - къщи за гости", value: ""}, %Frezu.Anchor{__meta__: #Ecto.Schema.Metadata<:loaded

This is part of error, rest is other list items truncated at some point.

There is purple messages above each error message

[debug] QUERY OK db=0.5ms
commit []

idi527 · February 20, 2018, 8:41am

should probably return {:ok, []}

def create(struct, params \\ %{}) do

I would avoid defaulting params to an empty map \\ %{} since later in code you are trying to access :mirror on it

|> Ecto.Changeset.put_assoc(:mirror, params.mirror)

Can you create a small project which would reproduce this error and upload it on github? It can just be these functions with hardcoded data (which causes problems).

hristonev · February 20, 2018, 9:07am

Yep, thanks that was part of the problem. Other one was that I was trying to insert via Repo.insert result from “create” function.
I can deploy project as is on github. Its learning project.

hristonev · February 20, 2018, 10:26am

This is my project. If you see some major stupidity please tell me to fix it.
In short you can create account, download and parse web sites.