Guide: How to enforce key sort order when JSON encoding a struct (using Jason or JSON)

Phoenix Forum Chat / Discussions
, , , ,
1

I’ve been working with some pretty large JSON objects, and the inherently unordered sorting caused by parsing a struct as a map makes it hard to find the keys that I am looking for when dealing with these large objects.

To resolve this, I figured out how to convert structs to JSON while preserving a desired sort order (e.g. alphabetical), by converting the structs to keyword lists before encoding them as JSON.

I’m dumping this info here in case it’s helpful for someone else. There may be errors that I am not aware of, but it has worked for me so far. (Please let me know if you find an issue!)

NOTE: The JSON spec (RFC 8259) defines the JSON key-value pairs as being unordered, but knowing this doesn’t make a large JSON object any easier for a person to read. This guide is intended to help the humans who have to read the JSON, not the machines.

Jason (Hex package)

Jason has a helper function to JSON-encode a keyword list. Here’s how to use it to encode a struct:

lib/your_project/some_struct.ex

defmodule YourProject.SomeStruct do
  use Ecto.Schema

  embedded_schema do
    field :hello, :string
    field :world, :string
  end

  defimpl Jason.Encoder do
    @doc "Sort keys alphabetically."
    def encode(struct, opts) do
      struct
      |> Map.from_struct()
      |> Enum.into([])
      |> Enum.sort()
      |> Jason.Encode.keyword(opts)
    end
  end
end

Try it out in IEx:

 iex(1)> %YourProject.SomeStruct{} |> Jason.encode!()
"{\"hello\":null,\"id\":null,\"world\":null}"
 iex(2)> |> Jason.decode!()
%{"hello" => nil, "id" => nil, "world" => nil}

JSON (Native Elixir module in version >=1.18.0)

The newer native JSON Elixir module does not appear to have the same convenience function offered by Jason, so we need to define it ourselves (inspiration taken from JSON.Encoder.Map):

lib/your_project/json_helpers.ex

defmodule YourProject.JSONHelpers do
  def encode_keyword_list!(value, encoder)

  def encode_keyword_list!([], _encoder), do: "{}"

  def encode_keyword_list!([{key, value}], encoder) do
    [?{, key(key, encoder), ?:, encoder.(value, encoder), ?}]
  end

  def encode_keyword_list!([{key, value} | tail], encoder) do
    [?{, key(key, encoder), ?:, encoder.(value, encoder) | next(tail, encoder)]
  end

  defp next([], _encoder), do: ~c"}"

  defp next([{key, value} | tail], encoder) do
    [?,, key(key, encoder), ?:, encoder.(value, encoder) | next(tail, encoder)]
  end

  defp key(key, encoder) when is_atom(key), do: encoder.(Atom.to_string(key), encoder)
  defp key(key, encoder) when is_binary(key), do: encoder.(key, encoder)
  defp key(key, encoder), do: encoder.(String.Chars.to_string(key), encoder)
end

Here’s the same struct from before, but tweaked to work with the native JSON module and our custom helper function:

lib/your_project/some_struct.ex

defmodule YourProject.SomeStruct do
  use Ecto.Schema

  embedded_schema do
    field :hello, :string
    field :world, :string
  end

  defimpl JSON.Encoder do
    @doc "Sort keys alphabetically."
    def encode(value, encoder) do
      value
      |> Map.from_struct()
      |> Enum.into([])
      |> Enum.sort()
      |> YourProject.JSONHelpers.encode_keyword_list!(encoder)
    end
  end
end

Try it out in IEx:

 iex(1)> %YourProject.SomeStruct{} |> JSON.encode!()
"{\"hello\":null,\"id\":null,\"world\":null}"
 iex(2)> |> JSON.decode!()
%{"hello" => nil, "id" => nil, "world" => nil}

Hopefully this is helpful to someone else out there!

4 Likes
2

The Jason library also has an OrdereObject module that can be used to preserve order: Jason.OrderedObject — jason v1.4.4

1 Like
3

Oh, right! I never figured out how to use that.

Can you provide an example?

1 Like
4

Here you go:

iex(1)> Mix.install [:jason]
:ok
iex(2)> map = %{a: 10, b: 3, c: 1, d: 100}
%{c: 1, a: 10, b: 3, d: 100}
iex(3)> map |> Enum.sort_by(&elem(&1,1)) |> Jason.OrderedObject.new() |> Jason.encode!(pretty: true) |> IO.puts()
{
  "c": 1,
  "b": 3,
  "a": 10,
  "d": 100
}
3 Likes