Hello,

After a following several get started tutorials and trying a few things, I’m seriously considering migrating my project to Ash framework.

Before I really start to migrate, I would like a clean view on how to test my app under Ash framework.

I’ve rode a few posts but not a lot talking about that:

• Testing Ash framework apps
• Ash Api testing setup (intermittent unrelated test failures in application)

And from the official doc:

• Testing With Postgres

I learned with Elixir community that tests are a very important part of an application, and I like the TDD approach even if I don’t follow it all the time. Tests remain essential in a development process, I don’t invent anything.

That’s why I create this post.

My goal is to regroup feedbacks from the community on how you handle the tests of your application, when using Ash framework, before than in-depth articles and tutorials are written on this subject.

I have my own questions to feed the subject:

Usage

1. Resource generation (fixture)

Within a classical Phoenix application, fixtures are written as follow:

defmodule MyApp.SupportFixtures do
  @moduledoc """
  This module defines test helpers for creating
  entities via the `MyApp.Support` context.
  """

  @doc """
  Generate a ticket.
  """
  def ticket_fixture(attrs \\ %{}) do
    {:ok, ticket} =
      attrs
      |> Enum.into(ticket_valid_attrs())
      |> MyApp.Support.create_ticket()

    ticket
  end

  def ticket_valid_attrs() do
    %{subject: "some subject"}
  end
end

Then you can test like that, and it works:

defmodule MyApp.SupportTest do
  use MyApp.DataCase

  alias MyApp.Support

  import MyApp.SupportFixtures

  @tag :phoenix
  describe "tickets" do
    test "list_tickets/0 returns all tickets" do
      ticket = ticket_fixture()
      assert Support.list_tickets() == [ticket]
    end
  end
end

Now with Ash Framework, generating the fixture can looks like:

defmodule MyApp.SupportFixtures do
  @moduledoc """
  This module defines test helpers for creating
  entities via the `MyApp.Support` context.
  """

  alias MyApp.Support
  alias MyApp.Support.Ticket

  @doc """
  Generate a ticket.
  """
  def ticket_fixture(attrs \\ %{}) do
    final_attrs = attrs |> Enum.into(ticket_valid_attrs())

      Ash.Changeset.for_create(Ticket, :create, final_attrs)
      |> Support.create!()
  end

  def ticket_valid_attrs() do
    %{subject: "some subject"}
  end
end

And the test could be:

defmodule MyApp.SupportTest do
  use MyApp.DataCase

  alias MyApp.Support

  import MyApp.SupportFixtures

  @tag :ash
  describe "tickets" do
    alias MyApp.Support.Ticket

    test "read returns all tickets" do
      ticket = ticket_fixture()

      assert Ash.Query.for_read(Ticket, :read) |> Support.read!() ==
               [ticket]
    end
  end
end

But this test will fail, with the following error message:
(because coloration doesn’t help here, the only difference here is on the metadata field)

Assertion with == failed
     code:  assert Ash.Query.for_read(Ticket, :read) |> Support.read!() == [ticket]
     left:  [
              %MyApp.Support.Ticket{
                __lateral_join_source__: nil,
                __meta__: #Ecto.Schema.Metadata<:loaded, "tickets">,
                __metadata__: %{
                  selected: [:id, :subject, :status, :representative_id]
                },
                __order__: nil,
                aggregates: %{},
                calculations: %{},
                id: "9ec80178-b60d-48f2-b86e-892300d6ba90",
                representative: #Ash.NotLoaded<:relationship, field: :representative>,
                representative_id: nil,
                status: :open,
                subject: "some subject"
              }
            ]
     right: [
              %MyApp.Support.Ticket{
                __lateral_join_source__: nil,
                __meta__: #Ecto.Schema.Metadata<:loaded, "tickets">,
                __metadata__: %{},
                __order__: nil,
                aggregates: %{},
                calculations: %{},
                id: "9ec80178-b60d-48f2-b86e-892300d6ba90",
                representative: #Ash.NotLoaded<:relationship, field: :representative>,
                representative_id: nil,
                status: :open,
                subject: "some subject"
              }
            ]

The basic solution I found to fix it is querying the ticket from the database after it’s creation inside the fixture:

defmodule MyApp.SupportFixtures do
  @moduledoc """
  This module defines test helpers for creating
  entities via the `MyApp.Support` context.
  """

  alias MyApp.Support
  alias MyApp.Support.Ticket

  @doc """
  Generate a ticket.
  """
  def ticket_fixture(attrs \\ %{}) do
    final_attrs = attrs |> Enum.into(ticket_valid_attrs())

    ticket =
      Ash.Changeset.for_create(Ticket, :create, final_attrs)
      |> Support.create!()

    Ash.Query.for_read(Ticket, :by_id, %{id: ticket.id})
    |> Support.read_one!()
  end

  def ticket_valid_attrs() do
    %{subject: "some subject"}
  end
end

But I find this solution is redundant. I wonder if anyone has a better solution for such a case. How would you handle the variations of the metadata field?

2. JSON schema assertion

Within a classical Phoenix App, each schema has it’s own controller to handle requests at the server level, and JSON module to render the data:
Controller

defmodule MyAppWeb.Api.Support.TicketController do
  use MyAppWeb, :controller

  alias MyApp.Support

  action_fallback MyAppWeb.Api.FallbackController

  def index(conn, _params) do
    tickets = Support.list_tickets()
    render(conn, :index, tickets: tickets)
  end
end

JSON view

defmodule MyAppWeb.Api.Support.TicketJSON do
  alias MyApp.Support.Ticket

  @doc """
  Renders a list of tickets.
  """
  def index(%{tickets: tickets}) do
    %{data: for(ticket <- tickets, do: data(ticket))}
  end

  def data(%Ticket{} = ticket) do
    %{
      id: ticket.id,
      subject: ticket.subject,
      status: ticket.status,
    }
  end
end

Then you can test it like that:

defmodule MyAppWeb.Api.Support.TicketControllerTest do
  use MyAppWeb.ConnCase

  import MyApp.SupportFixtures

  alias MyAppWeb.Api.Support.TicketJSON

  setup %{conn: conn} do
    {:ok, conn: put_req_header(conn, "accept", "application/json")}
  end

  setup [:create_and_set_api_key]

  describe "index" do
    setup [:create_ticket]

    @tag :phoenix
    test "lists all tickets", %{conn: conn, ticket: ticket} do
      conn = get(conn, ~p"/api/v1/support/tickets")

      assert json_response(conn, 200)["data"] == [
               ticket |> to_normalised_json()
             ]
    end

  defp create_ticket(_) do
    ticket = ticket_fixture()

    %{ticket: ticket}
  end

  defp to_normalised_json(data) do
    data
    |> TicketJSON.data()
    |> normalise_json()
  end
end

With normalise_json/1 defined in MyAppWeb.ConnCase module:

defmodule MyAppWeb.ConnCase do
  @moduledoc """ ... """
  use ExUnit.CaseTemplate

  import Plug.Conn

[...]
  @doc """
  Format data to be normalised JSON and pass asserts in tests.
  """
  def normalise_json(data) do
    data
    |> Jason.encode!()
    |> Jason.decode!()
  end
[...]
end

It makes the assertion easy with any kind of schema since we pass the created ticket to the same function used by the controller to render the data.

With AshJsonApi, rendered resources are autogenerated from the resource description, so how would you make the assertion easy with data coming from a newly created ticket for instance? It would be nice to be able to use the same function that is used by AshJsonApi.Controllers.Get to generate the JsonApi schema for instance, but I’m currently not used to how it works.

defmodule MyAppWeb.AshControllerTest do
  use MyAppWeb.ConnCase

  import MyApp.SupportFixtures

  defp create_ticket(_) do
    ticket = ticket_fixture()

    %{ticket: ticket}
  end

  setup %{conn: conn} do
    {:ok, conn: put_req_header(conn, "accept", "application/vnd.api+json")}
  end

  setup [:create_and_set_api_key]

  describe "index" do
    setup [:create_ticket]

    @tag :ash
    test "lists all tickets", %{conn: conn, ticket: ticket} do
      conn = get(conn, ~p"/ash/tickets")

      assert json_response(conn, 200)["data"] == "data"
    end
  end
end

This code renders an error message:

Assertion with == failed
     code:  assert json_response(conn, 200)["data"] == "data"
     left:  [%{"attributes" => %{"representative_id" => nil, "status" => "open", "subject" => "some subject"}, "id" => "4ed2d671-2961-4ffc-9435-40413862ecf4", "links" => %{}, "meta" => %{}, "relationships" => %{"representative" => %{"links" => %{}, "meta" => %{}}}, "type" => "ticket"}]
     right: "data"

I could copy/paste the left side and re-arrange it to populate values with variables such as "id" => ticket.id, but I’m looking for something that I don’t need to modify in tests each time I will bring modifications to the JSON rendered schema.

Design

1. Design Ash controllers tests

As we have seen in my previous examples, a common design for API controllers in Phoenix applications is to have:

• 1 Controller file per schema
• 1 JSON file per schema

With Ash, and more specifically AshJsonApi, we have:

• 1 generic (dependency level) Controller file per action
• 1 generic (dependency level) JSON generator → for all resources? (to clarify, my comprehension is not perfect on it for now)

So I wonder what would be the best strategy for tests:

1. create a ControllerTest file for each resource (equivalent to ControllerTest file for each schema which I currently have on Phoenix)
   OR
2. create a ControllerTest file for each action

While writing it, I think the first option is the best, as I find it logic to regroup tests by resources (it creates a better overview of what you can / cannot do with your resource’s API, just as you have an overview of each resource in their respective Ash.Resource file).

Maybe some of you came with a third approach that I haven’t thought of, feedbacks are more than welcome to share the best practices!

I put again my questions here:

Usage

1. How would you handle the variations of the metadata field?
2. How would you normalize the assigns with JSON renders in controllers tests?

Design

1. How do you organize your controller tests with Ash (AshJsonApi or AshGraphQL even if I’m not sure to use the second in the short term)