Naively, I’d do the following:

1. Convert expected outputs to keyword lists;
2. Sort the expected outputs by their keys;
3. Do the same with the actual outputs.

E.g.

def normalize_map(map) when is_map(map) do
  map |> Map.to_list() |> Enum.sort_by(fn {key, _val} -> key end)
end

test "stuff" do
  expected_result = %{"one" => 1, "two" => 2, "four" => 4}

  assert MyModule.tested_function() |> normalize_map()
    == normalize_map(expected_result)
end

You can also make a macro out of it i.e. assert_equal_map that calls normalize_map (or just inlines the implementation) on both of its arguments.

Basically, introduce and force determinism.

Unless I’m missing something, I don’t see how atoms vs strings are relevant here.

Testing on the string order of json keys is in general unreliable. The JSON spec makes no promise of key order, nor do Elixir maps. If you want to test the semantic equivalence of something you need to decode the JSON, not assert on a JSON string.

Ideally you simply don’t encode to a string at all and instead test:

assert %{"one" => 1, "two" => 2} == output

If it is unavoidably already encoded then just decode it:

assert %{"one" => 1, "two" => 2} == Jason.decode!(output)

Yes, you’re trying to compare JSON as strings, while standard clearly defines that order of keys in json maps may be different for the same structure.

You can try doing something like

assert %{one: 1, two: 2} == result

If those strings are larger than 64 bytes, than they’re most likely shared. Smaller strings are not shared. But still, with atoms you’re saving just a few kilobytes for huge maps

Unless I’m missing something, I don’t see how atoms vs strings are relevant here.

What I understand from the highlights of OTP 26, the change of ordering of keys in maps only happens when keys are atoms, but I might be misinterpreting this either :

In OTP 26, as an optimization for certain map operations, such as maps:from_list/1 , maps with atom keys are now sorted in a different order. The new order is undefined and may change between different invocations of the Erlang VM.

Testing on the string order of json keys is in general unreliable.

I completely agree. Exactly because we’ve been able to get away with doing it like this for a long time, I am trying to understand how to do it right now and looking for feedback from other folks, e.g. how is everyone testing their JSON responses.

Testing for a semantic equivalence is actually a good point

Ideally you simply don’t encode to a string at all and instead test:

Btw this made me realise that the test failures I am looking at are coming from tests of controller actions, where the default JSON encoding from Phoenix kicks in. When testing pure view modules, there doesn’t seem to be a problem, since MyView.render(my_thing) returns Elixir map verbatim, which can be asserted to match some reference map during a test