Asserting on the JSON response from a controller action in OTP26

Phoenix Forum Questions / Help
,
1

After upgrading to OTP26, we’re seeing a lot of failures related to the shape of JSON response. For example, given the view is constructed using atoms (which is how we do it), the following assertion easily fails:

output = Jason.encode!(%{one: 1, two: 2})
assert  ~s|{"one":1,"two":2}| = output

This seems in line with one of the highlights of the OTP26 release, e.g. order of atom keys in map.

Simple assertions like above are possible to re-write so that order doesn’t matter. However, in our case, we have assertions fail on significantly bigger JSONs which are extremely hard to re-write.

The above makes me question if we aren’t doing anything wrong here.

Maybe it doesn’t make sense to use atoms when constructing the view? :thinking: The reason we always did it that way was to not to pollute process memory with strings that are common - a problem which (supposedly?) doesn’t exist for the same atoms.

Maybe testing shape of the output is not a good idea in the first place? If so, then how do you test your views?

How are you people dealing with a situation similar to this?

1 Like
2

Naively, I’d do the following:

  1. Convert expected outputs to keyword lists;
  2. Sort the expected outputs by their keys;
  3. Do the same with the actual outputs.

E.g.

def normalize_map(map) when is_map(map) do
  map |> Map.to_list() |> Enum.sort_by(fn {key, _val} -> key end)
end

test "stuff" do
  expected_result = %{"one" => 1, "two" => 2, "four" => 4}

  assert MyModule.tested_function() |> normalize_map()
    == normalize_map(expected_result)
end

You can also make a macro out of it i.e. assert_equal_map that calls normalize_map (or just inlines the implementation) on both of its arguments.

Basically, introduce and force determinism.

1 Like
3

Unless I’m missing something, I don’t see how atoms vs strings are relevant here.

Testing on the string order of json keys is in general unreliable. The JSON spec makes no promise of key order, nor do Elixir maps. If you want to test the semantic equivalence of something you need to decode the JSON, not assert on a JSON string.

Ideally you simply don’t encode to a string at all and instead test:

assert %{"one" => 1, "two" => 2} == output

If it is unavoidably already encoded then just decode it:

assert %{"one" => 1, "two" => 2} == Jason.decode!(output)
3 Likes
4

Yes, you’re trying to compare JSON as strings, while standard clearly defines that order of keys in json maps may be different for the same structure.

You can try doing something like

assert %{one: 1, two: 2} == result
2 Likes
5

If those strings are larger than 64 bytes, than they’re most likely shared. Smaller strings are not shared. But still, with atoms you’re saving just a few kilobytes for huge maps

2 Likes
6

Unless I’m missing something, I don’t see how atoms vs strings are relevant here.

What I understand from the highlights of OTP 26, the change of ordering of keys in maps only happens when keys are atoms, but I might be misinterpreting this either :thinking::

In OTP 26, as an optimization for certain map operations, such as maps:from_list/1 , maps with atom keys are now sorted in a different order. The new order is undefined and may change between different invocations of the Erlang VM.

Testing on the string order of json keys is in general unreliable.

I completely agree. Exactly because we’ve been able to get away with doing it like this for a long time, I am trying to understand how to do it right now and looking for feedback from other folks, e.g. how is everyone testing their JSON responses.

Testing for a semantic equivalence is actually a good point :thinking: :+1:

Ideally you simply don’t encode to a string at all and instead test:

Btw this made me realise that the test failures I am looking at are coming from tests of controller actions, where the default JSON encoding from Phoenix kicks in. When testing pure view modules, there doesn’t seem to be a problem, since MyView.render(my_thing) returns Elixir map verbatim, which can be asserted to match some reference map during a test :+1: