OvermindDL1
Ecto `many_to_many` questions
The new(ish) project that I’m working on started with Ecto 2 so I had access to many_to_many, however I had issues actually using it so I ended up falling back to has_many for two reasons before, the first being that I needed the extra data that was stored on the join table, and second that I could not get it to work.
Now has_many is a read-only mapping, no being able to store back in to it, but that feels like it wastes code and effort that could be automatically managed, at least in more simple join tables without extra data, and many_to_many is touted at having writing capability. So now I have a model that does have a many_to_many mapping and also has no extra useful data on the join table (beyond timestamps for logging purposes), so with that, here is the main model:
defmodule MyServer.Tag do
use MyServer.Web, :model
schema "tags" do
field :name, :string
field :description, :string
many_to_many :parents, MyServer.Tag,
join_through: MyServer.Tag.Tag,
join_keys: [child_id: :id, parent_id: :id],
on_replace: :delete,
on_delete: :delete_all
timestamps()
end
end
And the join table:
defmodule MyServer.Tag.Tag do
use MyServer.Web, :model
@primary_key false
schema "tags_tags" do
belongs_to :child, MyServer.Tag, primary_key: true
belongs_to :parent, MyServer.Tag, primary_key: true
timestamps()
end
end
I was going through this a lot yesterday, refactoring, changing styles, and I am certain the above definitions are broken somehow. The concept is that there are Tags, Tags can also be tagged. For example any tag that is tagged with an ‘Issue’ tag can be used in the issue tracker on issues. Unsure if the style is useful over a simple string mapping but I’ve not done this style before and it seems it could be useful so…
Now, say a tag is being created, we get something like:
iex> tag_params = %{"description" => "Testing", "name" => "Test", "parents" => ["1"]}
%{"description" => "Testing", "name" => "Test", "parents" => ["1", "4"]}
Say from a web form, it should create a new tag that is also tagged with the tags with the IDs of 1 and 4 (the creation form gets a list of IDs that the current user are allowed to attach to and assume that I’ve already checked that 1 and 4 are allowed here too, I have a very fine-grained permission system). What would be the way to attach it? My immediate thought would be:
changeset = Tag.changeset_assoc(%Tag{}, tag_params)
Repo.insert(changeset)
Where changeset_assoc/2 and changeset/2 are:
def changeset(struct, params \\ %{}) do
struct
|> cast(params, [:name, :description])
|> validate_required([:name, :description])
|> unique_constraint(:name)
|> unique_constraint(:id, name: "tags_pkey")
end
def changeset_assoc(struct, params \\ %{}) do
struct
|> changeset(params)
|> cast_assoc(:parents) #, with: &changeset_as_assoc(&1, &2))
end
My immediate thought as to why that would work is because the primary_key on Tag is id of an integer, and I would think that it would be able to look up a prior-existing Tag with that ID (or error if it did not exist). However, instead I get this error:
** (exit) an exception was raised:
** (ArgumentError) argument error
:erlang.apply("1", :id, [])
(my_server) anonymous fn/1 in MyServer.ViewExtras.tag_input/4
(elixir) lib/enum.ex:1184: Enum."-map/2-lists^map/1-0-"/2
(my_server) web/view_extras.ex:34: MyServer.ViewExtras.tag_input/4
(my_server) web/templates/tag/form.html.eex:22: anonymous fn/2 in MyServer.TagView.form.html/1
(phoenix_html) lib/phoenix_html/form.ex:235: Phoenix.HTML.Form.form_for/4
(my_server) web/templates/tag/form.html.eex:1: MyServer.TagView."form.html"/1
(my_server) web/templates/tag/new.html.eex:3: MyServer.TagView."new.html"/1
(my_server) web/templates/layout/app.html.eex:79: MyServer.LayoutView."app.html"/1
(phoenix) lib/phoenix/view.ex:335: Phoenix.View.render_to_iodata/3
(phoenix) lib/phoenix/controller.ex:641: Phoenix.Controller.do_render/4
(my_server) web/controllers/tag_controller.ex:1: MyServer.TagController.action/2
(my_server) web/controllers/tag_controller.ex:1: MyServer.TagController.phoenix_controller_pipeline/2
(my_server) lib/my_server/endpoint.ex:1: MyServer.Endpoint.instrument/4
(my_server) lib/phoenix/router.ex:261: MyServer.Router.dispatch/2
(my_server) web/router.ex:1: MyServer.Router.do_call/2
(my_server) lib/my_server/endpoint.ex:1: MyServer.Endpoint.phoenix_pipeline/1
(my_server) lib/plug/debugger.ex:93: MyServer.Endpoint."call (overridable 3)"/2
(my_server) lib/my_server/endpoint.ex:1: MyServer.Endpoint.call/2
(plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
Uh, hrmm, well it tried to call “1” as a function, in that case I bet that it is actually assuming that it is a map, well lets try this as the changeset function instead:
parents_params = Map.get(tag_params, "parents", [])
parents_ids = Enum.map(parents_params, fn t ->
String.to_integer(t)
end)
parents = parents_ids |> Enum.map(fn t -> %{id: t} end)
changeset = Tag.changeset_assoc(%Tag{parents: []}, put_in(tag_params["parents"], parents))
And try it… and changeset has an error:
iex> changeset
#Ecto.Changeset<action: nil,
changes: %{description: "Testing", name: "Test",
parents: [#Ecto.Changeset<action: :insert, changes: %{},
errors: [name: {"can't be blank", []},
description: {"can't be blank", []}], data: #my_MyServer.Tag<>,
valid?: false>,
#Ecto.Changeset<action: :insert, changes: %{},
errors: [name: {"can't be blank", []},
description: {"can't be blank", []}], data: #my_MyServer.Tag<>,
valid?: false>]}, errors: [], data: #my_MyServer.Tag<>, valid?: false>}
Hmm, it seems that it is trying to insert new tags as parents instead of re-using existing ones, so I guess it does not try to lookup existing ones at all… So instead what if I lookup the Tags and put ‘those’ into the table, changing the entire changeset lookup into:
parents_params = Map.get(tag_params, "parents", [])
parents_ids = Enum.map(parents_params, fn t ->
String.to_integer(t)
end)
parents = Repo.all(from t in Tag, where: t.id in ^parents_ids)
changeset = Tag.changeset_assoc(%Tag{parents: []}, put_in(tag_params["parents"], parents))
And give this a try:
** (Ecto.CastError) expected params to be a map, got: `%my_MyServer.Tag{__meta__: #Ecto.Schema.Metadata<:loaded, "tags">, description: "Parent tag for Issues", id: 1, inserted_at: #Ecto.DateTime<2016-07-05 21:27:32>, name: "Issues", parents: #Ecto.Association.NotLoaded<association :parents is not loaded>, updated_at: #Ecto.DateTime<2016-07-05 21:27:32>}`
(ecto) lib/ecto/changeset.ex:326: Ecto.Changeset.do_cast/4
(my_server) web/models/tag.ex:22: my_MyServer.Tag.changeset/2
(ecto) lib/ecto/changeset/relation.ex:98: Ecto.Changeset.Relation.do_cast/5
(ecto) lib/ecto/changeset/relation.ex:284: Ecto.Changeset.Relation.map_changes/7
(ecto) lib/ecto/changeset.ex:533: Ecto.Changeset.cast_relation/4
(my_server) web/controllers/tag_controller.ex:28: my_MyServer.TagController.create/2
(my_server) web/controllers/tag_controller.ex:1: my_MyServer.TagController.action/2
(my_server) web/controllers/tag_controller.ex:1: my_MyServer.TagController.phoenix_controller_pipeline/2
(my_server) lib/my_server/endpoint.ex:1: my_MyServer.Endpoint.instrument/4
(my_server) lib/phoenix/router.ex:261: my_MyServer.Router.dispatch/2
(my_server) web/router.ex:1: my_MyServer.Router.do_call/2
(my_server) lib/my_server/endpoint.ex:1: my_MyServer.Endpoint.phoenix_pipeline/1
(my_server) lib/plug/debugger.ex:93: my_MyServer.Endpoint."call (overridable 3)"/2
(my_server) lib/my_server/endpoint.ex:1: my_MyServer.Endpoint.call/2
(plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
(cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4
So it really badly wants a map, not a struct, lets try casting the Tag into a map then:
parents_params = Map.get(tag_params, "parents", [])
parents_ids = Enum.map(parents_params, fn t ->
String.to_integer(t)
end)
parents = Repo.all(from t in Tag, where: t.id in ^parents_ids)
|> Enum.map(fn t -> Map.from_struct(t) end)
changeset = Tag.changeset_assoc(%Tag{parents: []}, put_in(tag_params["parents"], parents))
And try this:
** (exit) an exception was raised:
** (Ecto.ConstraintError) constraint error when attempting to insert struct:
* unique: tags_lower_name_index
If you would like to convert this constraint into an error, please
call unique_constraint/3 in your changeset and define the proper
constraint name. The changeset defined the following constraints:
* unique: tags_pkey
* unique: tags_name_index
(ecto) lib/ecto/repo/schema.ex:403: anonymous fn/4 in Ecto.Repo.Schema.constraints_to_errors/3
(elixir) lib/enum.ex:1184: Enum."-map/2-lists^map/1-0-"/2
(ecto) lib/ecto/repo/schema.ex:393: Ecto.Repo.Schema.constraints_to_errors/3
(ecto) lib/ecto/repo/schema.ex:193: anonymous fn/11 in Ecto.Repo.Schema.do_insert/4
(ecto) lib/ecto/association.ex:931: Ecto.Association.ManyToMany.on_repo_change/4
(ecto) lib/ecto/association.ex:231: anonymous fn/7 in Ecto.Association.on_repo_change/6
(elixir) lib/enum.ex:1623: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto) lib/ecto/association.ex:228: Ecto.Association.on_repo_change/6
(elixir) lib/enum.ex:1623: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto) lib/ecto/association.ex:194: Ecto.Association.on_repo_change/3
(ecto) lib/ecto/repo/schema.ex:527: Ecto.Repo.Schema.process_children/4
(ecto) lib/ecto/repo/schema.ex:595: anonymous fn/3 in Ecto.Repo.Schema.wrap_in_transaction/6
(ecto) lib/ecto/adapters/sql.ex:472: anonymous fn/3 in Ecto.Adapters.SQL.do_transaction/3
(db_connection) lib/db_connection.ex:973: DBConnection.transaction_run/4
(db_connection) lib/db_connection.ex:897: DBConnection.run_begin/3
(db_connection) lib/db_connection.ex:671: DBConnection.transaction/3
(my_server) web/controllers/tag_controller.ex:34: my_MyServer.TagController.create/2
(my_server) web/controllers/tag_controller.ex:1: my_MyServer.TagController.action/2
(my_server) web/controllers/tag_controller.ex:1: my_MyServer.TagController.phoenix_controller_pipeline/2
(my_server) lib/my_server/endpoint.ex:1: my_MyServer.Endpoint.instrument/4
Well it took it, but then it tried to create a new Tag with that data, having the database throw a constraint error, meaning I should change the :index constraint to have the name “tags_lower_name_index”, fixed, but this obviously does not work either. Maybe it will take a changeset instead, might have special code to handle that:
parents_params = Map.get(tag_params, "parents", [])
parents_ids = Enum.map(parents_params, fn t ->
String.to_integer(t)
end)
parents = Repo.all(from t in Tag, where: t.id in ^parents_ids)
|> Enum.map(fn t -> Ecto.Changeset.change(t) end)
changeset = Tag.changeset_assoc(%Tag{parents: []}, put_in(tag_params["parents"], parents))
And trying this:
** (Ecto.CastError) expected params to be a map, got: `#Ecto.Changeset<action: nil, changes: %{}, errors: [], data: #my_MyServer.Tag<>, valid?: true>`
(ecto) lib/ecto/changeset.ex:326: Ecto.Changeset.do_cast/4
(my_server) web/models/tag.ex:22: my_MyServer.Tag.changeset/2
(ecto) lib/ecto/changeset/relation.ex:98: Ecto.Changeset.Relation.do_cast/5
(ecto) lib/ecto/changeset/relation.ex:284: Ecto.Changeset.Relation.map_changes/7
(ecto) lib/ecto/changeset.ex:533: Ecto.Changeset.cast_relation/4
(my_server) web/controllers/tag_controller.ex:28: my_MyServer.TagController.create/2
(my_server) web/controllers/tag_controller.ex:1: my_MyServer.TagController.action/2
(my_server) web/controllers/tag_controller.ex:1: my_MyServer.TagController.phoenix_controller_pipeline/2
(my_server) lib/my_server/endpoint.ex:1: my_MyServer.Endpoint.instrument/4
(my_server) lib/phoenix/router.ex:261: my_MyServer.Router.dispatch/2
(my_server) web/router.ex:1: my_MyServer.Router.do_call/2
(my_server) lib/my_server/endpoint.ex:1: my_MyServer.Endpoint.phoenix_pipeline/1
(my_server) lib/plug/debugger.ex:93: my_MyServer.Endpoint."call (overridable 3)"/2
(my_server) lib/my_server/endpoint.ex:1: my_MyServer.Endpoint.call/2
(plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
(cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4
Nope. So how is this supposed to be able to set an existing tag as a parent of a tag? Maybe this ‘with:’ call on cast_assoc/4 might help, lets try to define something for it:
def changeset_assoc(struct, params \\ %{}) do
struct
|> changeset(params)
|> cast_assoc(:parents, with: &changeset_as_assoc(&1, &2))
end
def changeset_as_assoc(struct, params \\ %{}) do
my_MyServer.Repo.get!(from(my_MyServer.Tag), params.id) |> changeset()
end
After a quick experimenting I find that params is what each cell in the parents list is, so I give it the map from before since it will still not pass through the individual number strings from the web form (why do I have to massage these…):
parents_params = Map.get(tag_params, "parents", [])
parents_ids = Enum.map(parents_params, fn t ->
String.to_integer(t)
end)
parents = parents_ids |> Enum.map(fn t -> %{id: t} end)
changeset = Tag.changeset_assoc(%Tag{parents: []}, put_in(tag_params["parents"], parents))
And then give it a try:
** (exit) an exception was raised:
** (RuntimeError) attempting to cast or change association `parents` from `my_MyServer.Tag` that was not loaded. Please preload your associations before casting or changing the struct
(ecto) lib/ecto/changeset/relation.ex:65: Ecto.Changeset.Relation.load!/2
(ecto) lib/ecto/repo/schema.ex:424: anonymous fn/4 in Ecto.Repo.Schema.surface_changes/4
(elixir) lib/enum.ex:1623: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto) lib/ecto/repo/schema.ex:413: Ecto.Repo.Schema.surface_changes/4
(ecto) lib/ecto/repo/schema.ex:170: Ecto.Repo.Schema.do_insert/4
(ecto) lib/ecto/association.ex:931: Ecto.Association.ManyToMany.on_repo_change/4
(ecto) lib/ecto/association.ex:231: anonymous fn/7 in Ecto.Association.on_repo_change/6
(elixir) lib/enum.ex:1623: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto) lib/ecto/association.ex:228: Ecto.Association.on_repo_change/6
(elixir) lib/enum.ex:1623: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto) lib/ecto/association.ex:194: Ecto.Association.on_repo_change/3
(ecto) lib/ecto/repo/schema.ex:527: Ecto.Repo.Schema.process_children/4
(ecto) lib/ecto/repo/schema.ex:595: anonymous fn/3 in Ecto.Repo.Schema.wrap_in_transaction/6
(ecto) lib/ecto/adapters/sql.ex:472: anonymous fn/3 in Ecto.Adapters.SQL.do_transaction/3
(db_connection) lib/db_connection.ex:973: DBConnection.transaction_run/4
(db_connection) lib/db_connection.ex:897: DBConnection.run_begin/3
(db_connection) lib/db_connection.ex:671: DBConnection.transaction/3
(my_server) web/controllers/tag_controller.ex:34: my_MyServer.TagController.create/2
(my_server) web/controllers/tag_controller.ex:1: my_MyServer.TagController.action/2
(my_server) web/controllers/tag_controller.ex:1: my_MyServer.TagController.phoenix_controller_pipeline/2
Although the changeset shows no errors the actual Repo.insert call failed by saying the parents association is not loaded, yet how could it be loaded if the Tag is new and does not exist to ‘be’ loaded… Maybe it is talking of the one I load in changeset_as_assoc, lets change it to:
def changeset_as_assoc(struct, params \\ %{}) do
my_MyServer.Repo.get!(from(my_MyServer.Tag, preload: :parents), params.id) |> changeset()
end
And give this a try:
iex> Repo.insert(changeset)
{:error, #Ecto.Changeset<action: :insert,
changes: %{description: "Testing", name: "Test",
parents: [#Ecto.Changeset<action: :insert,
changes: %{description: "Parent tag for Issues", id: 1,
inserted_at: #Ecto.DateTime<2016-07-05 21:27:32>, name: "Issues",
updated_at: #Ecto.DateTime<2016-07-05 21:27:32>},
errors: [id: {"has already been taken", []}], data: #my_MyServer.Tag<>,
valid?: false>,
#Ecto.Changeset<action: :insert, changes: %{}, errors: [],
data: #my_MyServer.Tag<>, valid?: true>]}, errors: [],
data: #my_MyServer.Tag<>, valid?: false>}}
So, it still seems to be trying to insert a new parent instead of just joining it.
Stopping here for now as I kept having more of the same issues and had the same issues when trying to update a parent tag listing when both the child and parent tags already exist by updating the tag.parents list in a huge variety of ways, including put_assoc/4 all to no avail
So I have got to be a missing something obvious or documented somewhere or in the Ecto codebase that I must be missing, so could someone show me the code needed to insert a new tag with parents (it works if parents is empty) and to update an existing tag with parents, in all cases the parents already exist and cannot be dynamically created? >.>
Trending in Questions
Other Trending Topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #performance
- #security










First 10 of 18 Posts!
hubertlepicki
I would not try to do what you try to do at all. It all comes down to simplicity. Nested changesets, embedded models, updating associated models in general, validations spanning across multiple models… no, just no.
Why don’t you simply fist create post, then ensure tags exist, then create associations between tags, all manually instead of relying on the ORM magic to do it for you?
I know it’s not answer to your question, but sort of solution. I avoid such magical ORM features in Rails, do not plan to use them in Ecto for the very reason that you end up with complications like above.
OvermindDL1
I avoid such things elsewhere, I prefer raw SQL in many ways (one big reason I like the ecto query syntax is it is a near direct 1-to-1 mapping for the sql), but I am just trying to reason out how this is ‘supposed to’ work. Everywhere else I do manage the association tables manually in entirety. This is mostly a learning exercise to figure out how
many_to_manyis supposed to work.No posts here, the tags are used for marking various things in various areas, but for now as a test to learn how
many_to_many’s work I am trying to use it here. As I mentioned at the end of my post it is a similar issue if I try to update an existing Tag as well, same problems. I can get a Tag, get a list of other tags, and try to set that list as the parent tags for this first tag, same errors in the same ways whether using ID’s, changesets, maps, etc… So no, withmany_to_manyI still have the same issue whether updating an existing tag or making a new one. Feel free to substitute through-out the post where making a new Tag I pass in an existing instead, same things happen.hubertlepicki
I am with you on the SQL thing. One think I did notice about your code, is taht you do have
many_to_many :parentsbut don’t havemany_to_many :children. I am uncertain if that’s related at all to your problem, but it may be needed to define both sides of the relation, in the same model, when it’s self-referrencingOvermindDL1
Hmm, I really only need a
many_to_manyfrom children to parents and not the other way, but it is possible Ecto needs that… I’ll try it out probably during lunch.OvermindDL1
Yeah same issue with adding a children
many_to_manyrelation, ran through the battery of tests again and same results as the initial post.OvermindDL1
Tried more at work today and still failing. Has to be something obvious I’m missing. Anyone see what?
xadhoom
I spent an entire sunday afternoon trying to do a similar thing (and encurring in same errors).
And ended up doing as @hubertlepicki suggests, basically manually updating the join table (via an Ecto.Schema).
If I need to update the relationship, without updating/inserting the linked schema, I just delete_all from the join table using the parent id and rebuild every single association, something like:
In this example each sensor has a many to many relation with partitions, and PartitionSensor is the join model on top if the join table.
Then wrap everything (along with your other updates, if any) under an Ecto.Repo.trasaction to avoid partial operations.
Works for me, but really I’m curious to understand if is the propery way, or there’s a better way…
josevalim
I am sorry for the confusion on this one. Coincidentally, I think you are having those issues because you are expecting Ecto to do too much for you.
cast_assoc expects all the data to be in parameters. Think about it as an operation that merges the parameters into the Elixir data, seeing what is new and what changed, and generating proper commands.
If you are creating data on the side, then it is better to tell the changeset: this is the new data, deal with it. The solution would rather be something along those lines inside the changeset function:
I do agree with @hubertlepicki though. Many times it is simpler to just perform multiple queries. Constructs such as Ecto.Multi makes it straight-forward to define multiple commands that will run in a transaction without the complexities in handling roll backs and without the hierarchy that comes with changesets.
I will see if we can make the documentation better on this.
xadhoom
mmmh so If my params contains the associations as a list of associated items with the relevant ids and so on, Ecto should be able to check that the associated items already exists and just update the join table, by removing and adding “links” on it ?
josevalim
Yes.
Last Post!
krishna_vaguelyright
This whole thread is great. I have broken my head on this for over a week & couldnt get it to work and finally went ahead with the has_many version of it & the frontend team doubting over my skills that I couldn’t get a simple many_to_many work.
I have tried to use
cast_assoc,put_assoceverything that I could find online to make it work searched for github repos for examples. Ecto tries to insert the associated enteries everytime!!! with id, structs, changeset, everything…If somebody could point me to a working example. That would be great! Thanks!