Authorization - Roles Map?

I am building an application that will have many users and groups. Each user can have one or more roles per group. My plan is to store roles as a map, or jsonb in the database, and I’m using binary ids across the board. In my current plan, a user’s roles will look like this:

  <group_id>: [:owner],
  <another_group_id>: [:moderator],
  <yet_another_group_id>: [:vip],

Permissions will be defined elsewhere. Currently, the %User{} schema defines roles as field :roles, :map, default: %{}. This works, but I’m not sure it’s robust enough.

My question is, is it possible to create associations to groups using this approach? Do you see any gotchas or pitfalls? Feel free to poke holes. :face_with_monocle: