How to decode a JSON object into a keyword list while preserving the same key order?

Questions / Help
1

How to decode a JSON object into a, for instance, keyword list while preserving the same key order that the JSON string uses?

Elixir maps lose this information because of the underlaying Erlang map implementation.

2

According to the JSON specification, JSON objects are unordered. So if you get the same ordering, it is entirely coincidental or an implementation detail of the underlying library. But you can not rely on ordering of the entries there since they are, by specification, not ordered.

You may want to considering using a list instead if the ordering is absolutely required.

3 Likes
3

Precisely this. JSON specifies that objects are unordered but array’s are ordered, so if order is important then it should be an array, not an object.

4

I know specification says JSON objects are unordered but sometimes preserving the order can be useful.

Take for instances a situation where you want to read an external JSON file, apply some small changes and write it back preserving the order so to not create large diffs (the file can be under version control).

For instance in python you can do this with this code:

import json
from collections import OrderedDict
json.load(input_json, object_pairs_hook=OrderedDict)

In Elixir I am only able to write JSON objects in a custom order by using Keyword instead of Map but I haven’t find a way of reading a JSON object into a Keyword.

5

In jason it seems like objects are actually decoded into a list until the whole object is read and only at that moment the list is converted to a map:

But given jason seems to want to be spec compliant I’m not sure it would accept changing that.

6

Thanks, I am able to use this :jason implementation detail to get what I want.

It is a very far from optimal solution, surely not performant and relies on the order in which keys are decoded by Jason but it seems to work for my needs (it’s more like a script than something that should be done a lot of times per second).

I put it here just for people that may have my same requirements but I wrote it without optimizing it so be kind with your comments :smiley:

defmodule StringKeyword do
  defstruct data: []

  @spec decode!(binary) :: any
  def decode!(string) when is_bitstring(string) do
    Process.put(:key_count, 0)
    
    string
    |> Jason.decode!(
      keys: fn key ->
        count = Process.get(:key_count)
        Process.put(:key_count, count + 1)
        {count, key}
      end
    )
    |> new()
  end

  defp new(conf) when is_map(conf) do
    data =
      conf
      |> Map.to_list()
      |> Enum.sort_by(fn {{idx, _}, _} -> idx end)
      |> Enum.map(fn {{_, k}, v} -> {k, new(v)} end)

    %__MODULE__{data: data}
  end

  defp new(list) when is_list(list), do: Enum.map(list, &new/1)

  defp new(value), do: value
end

defimpl Jason.Encoder, for: StringKeyword do
  def encode(%{data: value}, opts), do: Jason.Encode.keyword(value, opts)
end

defmodule Sorted do
  def run(input, output) do
    input
    |> File.read!()
    |> StringKeyword.decode!()
    |> Jason.encode!(pretty: [indent: "    "])
    |> (&File.write!(output, &1)).()
  end
end

Edited as per @LostKobrakai suggestion

7

How do you write the loaded json back? Wouldn’t the order of the keys be not deterministic in this step?

8

By using Jason.Encode.keyword/2. This function preserves the order of the Keyword in encoding it.

9

Why combine the data for the key into a string just to need a regex to separate it again. You can have a key like {count, key} in elixir maps.

2 Likes
10

This is exactly what I mean, I wrote this “without thinking” to much, just to see if it can work. Thanks for your suggestion!

1 Like
11

By the way I had a problem where I had to get the json ordered.

I found out we can use Jason.decode(json_string, %{objects: :ordered_objects})

2 Likes
12

Exactly the objects: :ordered_objects is the way to solve this, no need for hacky solutions. It decodes into a Jason.OrderedObject struct that implements Access and Enumerable and can be used similarly to a keyword list. You can also access the .values field for the actual raw keyword. The ordered object struct also encodes into JSON with the specified order.

And yes, while JSON is explicitly not ordered and relying on order is not conforming, there’s not much you can do usually, when interacting with APIs where the order does matter

13

Needed to have JSON with sorted keys, so threw together a simple library based on Jason.OrderedObject.

https://hex.pm/packages/stable_jason