How to represent Keyword list in Ecto / Ecto.Schema

@silverdr

Would a list of single KV maps be an option? It would probably be serializable to JSON automatically. It also doesn’t appear to be much more expensive than keyword lists.

iex(3)> :erts_debug.size [%{a: :b}, %{c: :d}]
16
iex(4)> :erts_debug.size [a: :b, c: :d]
10
iex(5)> :erts_debug.size Enum.map(1..100, fn i -> %{i => i} end)
800
iex(6)> :erts_debug.size Enum.map(1..100, fn i -> {i, i} end)
500

Does it matter what the data stored in the JSON looks like, do you need to issue db queries againt it?

If not then an option could be to unzip the KV list into a 2d array in the changeset operation before storing/updating, and post-process the repo query to zip it back up when taking it out again.

fruits = ["apple", "banana", "orange"]
counts = [3, 1, 6]
Enum.zip(fruits, counts)
[{"apple", 3}, {"banana", 1}, {"orange", 6}]

from zipping cheatsheet

Since you’re contemplating a Keyword list I assume your data structure has known atom fields, in which case you can also transform the keys into known atoms during the zipping, otherwise you’d run into atom table issues.

But if keys are known and have set order, then you could just store the values as a sparse array and rehydrate your Keyword list from there

This is probably a good candidate for a custom Ecro type, but without knowing more about your data structure it’s hard to give a more definitive answer.

That’s how I initially did it but didn’t like what I had. Not even because of the cost you referred to. But rather because accessing values by keys becomes… well… ugly I guess is the word.

Say I have something like:

[a: 12, b: 102, c: 9]

vs.

[%{a: 12}, %{b: 102}, %{c: 9}]

How do I quickly get the value for b: (or whichever) key?

iex(1)> kv = [%{a: 12}, %{b: 102}, %{c: 9}]
[%{a: 12}, %{b: 102}, %{c: 9}]
iex(2)> Enum.find_value(kv, fn kv -> kv[:b] end)
102

Although I hope(d) for an array of JSON “objects” what you say is probably the easier, possible scenario. Yes, the keys come from a known, limited set and splitting keys and values into separate arrays would allow for relatively easy pre/post processing. “Not good, not terrible”

Heh… yes, there’s already a function which hides all the ugliness under its hood! But at least it doesn’t hurt the eyes. Thank you, this might be an option in this case - it’s not a traffic heavy spot in the application so let’s see

Because I can’t leave a thread flapping around in my head I tried things out and it’s not easy to use the existing Ecto functionality to store an array of arrays, but you can approximate it with embedded structures and functions to marshall between your data. For example:

defmodule KWList do
  use Ecto.Schema
  import Ecto.Changeset

  schema "kwlists" do
    embeds_many :kw_pairs, Pair do
      field :key, :string
      field :value, :integer
    end

    field :data, :any, virtual: true

    timestamps(type: :utc_datetime)
  end

  @doc false
  def changeset(kw_list, attrs) do
    attrs = data_to_pairs(attrs)

    kw_list
    |> cast(attrs, [])
    |> cast_embed(:kw_pairs, with: &pairs_changeset/2)
  end

  def data_to_pairs(attrs) do
    {data_key, pairs_key} = typed_keys(attrs)

    pairs = attrs[data_key] || []

    kw_pairs =
      for {k, v} <- pairs do
        %{key: k, value: v}
      end

    Map.put(attrs, pairs_key, kw_pairs)
  end

  def typed_keys(attrs) do
    case Enum.take(attrs, 1) do
      [{k, _}] when is_binary(k) -> {"data", "kw_pairs"}
      _ -> {:data, :kw_pairs}
    end
  end

  def pairs_changeset(schema, attrs) do
    schema
    |> cast(attrs, [:key, :value])
  end

  def convert_keys(%KWList{kw_pairs: pairs} = schema) when is_list(pairs) do
    data =
      for %{key: k, value: v} <- pairs do
        {k, v}
      end

    %{schema | data: data}
  end

  def convert_keys(%KWList{} = schema) do
    schema
  end
end

edit: Note I fixed a bug in the typed_keys case statement, match should be [{k, _}] when is_binary(k), not {k, _} when ...

iex(2)> c = KWList.changeset(%KWList{}, %{data: [{"d", 2}, {"b", 3}]})
#Ecto.Changeset<
  action: nil,
  changes: %{
    kw_pairs: [
      #Ecto.Changeset<
        action: :insert,
        changes: %{value: 2, key: "d"},
        errors: [],
        data: #KWList.Pair<>,
        valid?: true
      >,
      #Ecto.Changeset<
        action: :insert,
        changes: %{value: 3, key: "b"},
        errors: [],
        data: #KWList.Pair<>,
        valid?: true
      >
    ]
  },
  errors: [],
  data: #KWList<>,
  valid?: true
>
iex(3)> Repo.insert(c)
...logs...
{:ok,
 %KWList{
   __meta__: #Ecto.Schema.Metadata<:loaded, "kwlists">,
   id: 2,
   kw_pairs: [
     %KWList.Pair{
       id: "fc0e5ca8-a369-4b48-ae15-a80834bbb122",
       key: "d",
       value: 2
     },
     %KWList.Pair{
       id: "166f7beb-4830-4506-9070-3e4a173aaf89",
       key: "b",
       value: 3
     }
   ],
   data: nil,
   inserted_at: ~U[2024-05-04 08:21:47Z],
   updated_at: ~U[2024-05-04 08:21:47Z]
 }}


iex(4)> Repo.all(KWList) |> Enum.map(&KWList.convert_keys/1)
[debug] QUERY OK source="kwlists" db=6.4ms queue=3.6ms idle=1924.7ms
SELECT k0."id", k0."kw_pairs", k0."inserted_at", k0."updated_at" FROM "kwlists" AS k0 []
↳ :elixir.eval_external_handler/3, at: src/elixir.erl:405
[
  %KWList{
    __meta__: #Ecto.Schema.Metadata<:loaded, "kwlists">,
    id: 2,
    kw_pairs: [
      %KWList.Pair{
        id: "fc0e5ca8-a369-4b48-ae15-a80834bbb122",
        key: "d",
        value: 2
      },
      %KWList.Pair{
        id: "166f7beb-4830-4506-9070-3e4a173aaf89",
        key: "b",
        value: 3
      }
    ],
    data: [{"d", 2}, {"b", 3}],
    inserted_at: ~U[2024-05-04 08:21:47Z],
    updated_at: ~U[2024-05-04 08:21:47Z]
  }

And here’s how it looks in the db:

# \d kwlists
                                          Table "public.kwlists"
   Column    |              Type              | Collation | Nullable |               Default               
-------------+--------------------------------+-----------+----------+-------------------------------------
 id          | bigint                         |           | not null | nextval('kwlists_id_seq'::regclass)
 kw_pairs    | jsonb                          |           |          | 
 inserted_at | timestamp(0) without time zone |           | not null | 
 updated_at  | timestamp(0) without time zone |           | not null | 

# select kw_pairs from kwlists;
                                   kw_pairs                                   
-----------------------------------------------------------------------------------
 [{"id": "uuid1", "key": "d", "value": 2}, {"id": "uuid2", "key": "b", "value": 3}]

Or just:

schema "kwlist" do
  field :keys, {:array, :string}
  field :values, {:array, :integer}
end

and always make sure both arrays are mutated at the same time.

Though converting to known Keyword list with atoms makes traversing to find data much easier than writing your own reducers and manual loops.

With this solution, you could have a scenario where there are more keys than values and vice versa, right?

Love the attitude - highly appreciated!

Huh, yes - I already was halfway there with array of maps but that seems like an overkill for a theoretically simple problem, doesn’t it?

BTW in Ruby, starting with a don’t remember exactly which version (2.0?) the guys made Ruby hashmaps always preserve the order of keys. That one step made this class of problems as here simply disappear.