Queries (within validations) potentially behave differently in ex_unit tests

Ash Forum Ash Questions / Help
, ,
1

I’ve got two functions that run the same code, one inside an ex_unit test, and one just in a Livebook cell. They are both designed to test that my validation is working, so both should return an error. But the ex_unit one fails the test (of not returning an error).

# inside ex_unit, inside a Livebook cell
ExUnit.start(auto_run: false)

defmodule BoxOfficeFeaturesTest do
  use ExUnit.Case, async: false
  import BoxOffice.Core.Test

  setup do
    BoxOffice.Core.Test.reset()
  end

  describe "Reservations" do
    test "a reserved seat cannot be reserved again for the same performance" do
      create_seating_chart()
      [seat] = create_seats().records
      populate_seating_chart()
      create_production()
      performance = create_performance()

      assert {:ok, _} =
               BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)

      assert {:error, _} =
               BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)
    end
  end
end

results = ExUnit.run()

CleanShot 2024-01-19 at 17.23.15@2x
CleanShot 2024-01-19 at 17.23.15@2x1200×485 37.1 KB

# Livebook cell
defmodule WhoTestsTheTesters do
  import BoxOffice.Core.Test

  def run() do
    create_seating_chart()
    [seat] = create_seats().records
    populate_seating_chart()
    create_production()
    performance = create_performance()

    {:ok, _} =
      BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)

    {:error, _} =
      BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)
  end
end

BoxOffice.Core.Test.reset()
WhoTestsTheTesters.run()

CleanShot 2024-01-19 at 17.24.27@2x
CleanShot 2024-01-19 at 17.24.27@2x1334×630 45.7 KB

The latter is not throwing because it (correctly) matches {:error, _}. Just returns the error value.

Any idea why these two things would behave differently? Here’s the resource under test. In particular, I’ve discovered that previous_reservation is always [] in the test scenario:

defmodule BoxOffice.Core.Reservation do
  require Ash.Query

  use Ash.Resource,
    data_layer: Ash.DataLayer.Ets

  actions do
    defaults([:read, :update])

    # On creation set the patron by providing the id
    create(:make, do: accept([:patron_id, :performance_id, :seat_id]))
  end

  attributes do
    uuid_primary_key(:id)

    create_timestamp(:created_at)
    update_timestamp(:updated_at)
  end

  relationships do
    belongs_to(:patron, BoxOffice.Core.Patron) do
      # Set to writable so you can directly set the patron_id inside the :open action
      attribute_writable?(true)
    end

    belongs_to(:seat, BoxOffice.Core.Seat) do
      attribute_writable?(true)
    end

    belongs_to(:performance, BoxOffice.Core.Performance) do
      attribute_writable?(true)
    end
  end

  code_interface do
    define_for(BoxOffice.Core)

    define(:make, args: [:patron_id, :performance_id, :seat_id])
  end

  identities do
    # pre_check_with might only be necessary for ETS data layer.
    # see https://hexdocs.pm/ash/identities.html#pre-checking
    identity(:reservation_identity, [:seat_id, :performance_id], pre_check_with: BoxOffice.Core)
  end

  validations do
    validate(fn changeset ->
      seat_id = Ash.Changeset.get_attribute(changeset, :seat_id)
      performance_id = Ash.Changeset.get_attribute(changeset, :performance_id)

      previous_reservation =
        __MODULE__
        |> Ash.Query.filter(performance_id == ^performance_id and seat_id == ^seat_id)
        |> BoxOffice.Core.read!()
        |> dbg

      case previous_reservation do
        [] -> :ok
        [_] -> {:error, field: :seat_id, message: "This seat is already reserved"}
      end
    end)
  end
end
2

Have you added this to your test config in test.exs?

config :ash, :disable_async?, true
3

Seems like that could be it, because when I eliminate all the other tests, the one in question passes. But I am in Livebook and am not sure how to translate that config. I mean, I’ve done configs in Livebook before, but I also see some put_env stuff happening with Ash that I don’t understand. So I tried both.

I appreciate your help, and I’ll probably keep digging when Friday rolls around again. But if you are curious, or overly-generous with your time, here’s the entire Livebook:

# Ticketing core

```elixir
# Copy from Ash Livebook Tutorial
Application.put_env(:ash, :validate_api_resource_inclusion?, false)
Application.put_env(:ash, :validate_api_config_inclusion?, false)

# Attempt to make tests pass
Application.put_env(:ash, :disable_async?, true)

Mix.install([:ash, :kino, {:tz, "~> 0.26.5"}],
  config: [
    elixir: [time_zone_database: Tz.TimeZoneDatabase],
    # Separate attempt to make tests pass
    ash: [disable_async?: true]
  ],
  # Copy from Ash Livebook Tutorial
  consolidate_protocols: false
)
```

## Core definitions in Ash

```elixir
defmodule BoxOffice.Core.Patron do
  use Ash.Resource,
    data_layer: Ash.DataLayer.Ets

  actions(do: defaults([:create, :read, :update, :destroy]))

  attributes do
    uuid_primary_key(:id)
    attribute(:name, :string)

    create_timestamp(:created_at)
    update_timestamp(:updated_at)
  end

  # Added the inverse relationship of belongs_to in the reservation.
  # This way we can reference :reservations inside the aggregates.
  relationships do
    has_many(:reservations, BoxOffice.Core.Reservation)
  end

  code_interface do
    define_for(BoxOffice.Core)

    define(:create, args: [:name])
  end

  # <- Add the aggregates here
  aggregates do
    count(:count_of_reservations, :reservations)
  end
end

defmodule BoxOffice.Core.Seat do
  use Ash.Resource,
    data_layer: Ash.DataLayer.Ets

  actions(do: defaults([:create, :read, :update, :destroy]))

  attributes do
    uuid_primary_key(:id)
    attribute(:name, :string)

    create_timestamp(:created_at)
    update_timestamp(:updated_at)
  end

  relationships do
    has_many(:reservations, BoxOffice.Core.Reservation)

    many_to_many :seating_charts, BoxOffice.Core.SeatingChart do
      through(BoxOffice.Core.SeatSeatingChart)
      source_attribute_on_join_resource(:seat_id)
      destination_attribute_on_join_resource(:seating_chart_id)
    end
  end

  code_interface do
    define_for(BoxOffice.Core)

    define(:create, args: [:name])
  end
end

defmodule BoxOffice.Core.SeatSeatingChart do
  use Ash.Resource,
    data_layer: Ash.DataLayer.Ets

  relationships do
    belongs_to(:seat, BoxOffice.Core.Seat, primary_key?: true, allow_nil?: false)
    belongs_to(:seating_chart, BoxOffice.Core.SeatingChart, primary_key?: true, allow_nil?: false)
  end

  actions do
    defaults([:create, :read, :update, :destroy])
  end
end

defmodule BoxOffice.Core.SeatingChart do
  use Ash.Resource,
    data_layer: Ash.DataLayer.Ets

  actions do
    defaults([:create, :read, :update, :destroy])

    update :add_seat do
      accept([])
      argument(:seat_id, :uuid, allow_nil?: false)
      change(manage_relationship(:seat_id, :seats, type: :append))
    end
  end

  attributes do
    uuid_primary_key(:id)
    attribute(:name, :string)

    create_timestamp(:created_at)
    update_timestamp(:updated_at)
  end

  relationships do
    many_to_many :seats, BoxOffice.Core.Seat do
      through(BoxOffice.Core.SeatSeatingChart)
      source_attribute_on_join_resource(:seating_chart_id)
      destination_attribute_on_join_resource(:seat_id)
    end

    has_many(:performances, BoxOffice.Core.Performance)
  end

  code_interface do
    define_for(BoxOffice.Core)

    define(:create, args: [:name])
    define(:add_seat, args: [:seat_id])
  end

  aggregates do
    count(:count_of_seats, :seats)
  end
end

defmodule BoxOffice.Core.Production do
  use Ash.Resource,
    data_layer: Ash.DataLayer.Ets

  attributes do
    uuid_primary_key(:id)
    attribute(:title, :string)

    create_timestamp(:created_at)
    update_timestamp(:updated_at)
  end

  actions(do: defaults([:create, :read, :update, :destroy]))

  relationships do
    has_many(:performances, BoxOffice.Core.Performance)
  end

  code_interface do
    define_for(BoxOffice.Core)

    define(:create, args: [:title])
  end
end

defmodule BoxOffice.Core.Performance do
  use Ash.Resource,
    data_layer: Ash.DataLayer.Ets

  attributes do
    uuid_primary_key(:id)
    attribute(:datetime, :datetime)

    create_timestamp(:created_at)
    update_timestamp(:updated_at)
  end

  actions(do: defaults([:create, :read, :update, :destroy]))

  relationships do
    belongs_to(:production, BoxOffice.Core.Production) do
      attribute_writable?(true)
    end

    belongs_to(:seating_chart, BoxOffice.Core.SeatingChart) do
      attribute_writable?(true)
    end

    has_many(:reservations, BoxOffice.Core.Reservation)
  end

  code_interface do
    define_for(BoxOffice.Core)

    define(:create, args: [:production_id, :datetime, :seating_chart_id])
  end

  aggregates do
    count(:count_of_reserved_seats, :reservations)
  end

  calculations do
    calculate(
      :count_of_unreserved_seats,
      :integer,
      expr(seating_chart.count_of_seats - count_of_reserved_seats)
    )
  end
end

defmodule BoxOffice.Core.Reservation do
  require Ash.Query

  use Ash.Resource,
    data_layer: Ash.DataLayer.Ets

  actions do
    defaults([:read, :update])

    # On creation set the patron by providing the id
    create(:make, do: accept([:patron_id, :performance_id, :seat_id]))
  end

  attributes do
    uuid_primary_key(:id)

    create_timestamp(:created_at)
    update_timestamp(:updated_at)
  end

  relationships do
    belongs_to(:patron, BoxOffice.Core.Patron) do
      # Set to writable so you can directly set the patron_id inside the :open action
      attribute_writable?(true)
    end

    belongs_to(:seat, BoxOffice.Core.Seat) do
      attribute_writable?(true)
    end

    belongs_to(:performance, BoxOffice.Core.Performance) do
      attribute_writable?(true)
    end
  end

  code_interface do
    define_for(BoxOffice.Core)

    define(:make, args: [:patron_id, :performance_id, :seat_id])
  end

  identities do
    # pre_check_with might only be necessary for ETS data layer.
    # see https://hexdocs.pm/ash/identities.html#pre-checking
    identity(:reservation_identity, [:seat_id, :performance_id], pre_check_with: BoxOffice.Core)
  end

  validations do
    validate(fn changeset ->
      seat_id = Ash.Changeset.get_attribute(changeset, :seat_id)
      performance_id = Ash.Changeset.get_attribute(changeset, :performance_id)

      previous_reservation =
        __MODULE__
        |> Ash.Query.filter(performance_id == ^performance_id and seat_id == ^seat_id)
        |> BoxOffice.Core.read!()
        |> dbg

      case previous_reservation do
        [] -> :ok
        [_] -> {:error, field: :seat_id, message: "This seat is already reserved"}
      end
    end)
  end
end

defmodule BoxOffice.Core do
  use Ash.Api

  resources do
    resource(BoxOffice.Core.Reservation)
    resource(BoxOffice.Core.Patron)
    resource(BoxOffice.Core.Production)
    resource(BoxOffice.Core.Performance)
    resource(BoxOffice.Core.Seat)
    resource(BoxOffice.Core.SeatingChart)
    resource(BoxOffice.Core.SeatSeatingChart)
  end
end

require Ash.Query
```

## Test Setup

<!-- livebook:{"reevaluate_automatically":true} -->

```elixir
defmodule BoxOfficeCoreTest.Helpers do
  def reset() do
    BoxOffice.Core
    |> Ash.Api.Info.resources()
    |> Enum.reject(fn resource -> :ets.whereis(resource) == :undefined end)
    |> Enum.each(fn resource -> :ets.delete_all_objects(resource) end)
  end

  def create_patron(name \\ "Sally") do
    BoxOffice.Core.Patron.create!(name)
  end

  def create_production(name \\ "Some Great Show") do
    BoxOffice.Core.Production.create!(name)
  end

  def create_seating_chart(name \\ "main chart") do
    BoxOffice.Core.SeatingChart.create!(name)
  end

  def create_seats(names \\ ["A1"]) when is_list(names) do
    names
    |> Enum.map(fn name -> %{name: name} end)
    |> BoxOffice.Core.bulk_create!(BoxOffice.Core.Seat, :create, return_records?: true)
  end

  def create_performance(date \\ DateTime.from_naive!(~N[2024-01-01 20:00:00], "America/Chicago")) do
    [production] =
      BoxOffice.Core.Production
      |> Ash.Query.sort(created_at: :desc)
      |> Ash.Query.limit(1)
      |> BoxOffice.Core.read!()

    [seating_chart] =
      BoxOffice.Core.SeatingChart
      |> Ash.Query.sort(created_at: :desc)
      |> Ash.Query.limit(1)
      |> BoxOffice.Core.read!()

    create_performance(production, seating_chart, date)
  end

  def create_performance(
        production,
        seating_chart,
        date \\ DateTime.from_naive!(~N[2024-01-01 20:00:00], "America/Chicago")
      ) do
    BoxOffice.Core.Performance.create!(production.id, date, seating_chart.id)
  end

  def populate_seating_chart() do
    [seating_chart] =
      BoxOffice.Core.SeatingChart
      |> Ash.Query.sort(created_at: :desc)
      |> Ash.Query.limit(1)
      |> BoxOffice.Core.read!()

    seats =
      BoxOffice.Core.Seat
      |> Ash.Query.sort(created_at: :desc)
      |> BoxOffice.Core.read!()

    populate_seating_chart(seating_chart, seats)
  end

  def populate_seating_chart(%BoxOffice.Core.SeatingChart{} = seating_chart, seats) do
    seats
    |> Enum.map(fn seat ->
      BoxOffice.Core.SeatingChart.add_seat!(seating_chart, seat.id)
    end)
  end
end
```

## Test

<!-- livebook:{"reevaluate_automatically":true} -->

```elixir
ExUnit.start(auto_run: false)

defmodule BoxOfficeCoreTest do
  use ExUnit.Case
  import BoxOfficeCoreTest.Helpers

  setup do
    BoxOfficeCoreTest.Helpers.reset()
  end

  describe "Reservations" do
    test "we can reserve a seat" do
      create_seating_chart()
      [seat] = create_seats().records
      populate_seating_chart()
      create_production()
      performance = create_performance()

      assert {:ok, _} =
              BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)
    end

    test "we can query a reservation" do
      create_seating_chart()
      [seat] = create_seats().records
      populate_seating_chart()
      create_production()
      performance = create_performance()

      assert {:ok, reservation} =
              BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)

      assert [_] =
              BoxOffice.Core.Reservation
              |> Ash.Query.filter(id == ^reservation.id)
              |> BoxOffice.Core.read!()
    end

    test "a reserved seat cannot be reserved again for the same performance" do
      create_seating_chart()
      [seat] = create_seats().records
      populate_seating_chart()
      create_production()
      performance = create_performance()

      assert {:ok, _} =
              BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)

      assert {:error, _} =
              BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)
    end

    test "a seat outside the performance's chart is not valid when reserving" do
      create_seating_chart()
      create_seats()
      populate_seating_chart()
      create_production()
      performance = create_performance()

      [seat] = create_seats(["Z99"]).records

      assert {:error, _} =
              BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)
    end

    test "we can see how many reservations exist" do
      create_seating_chart()

      [seat | _] = create_seats(["1", "2"]).records

      populate_seating_chart()
      create_production()
      performance = create_performance()

      performance =
        BoxOffice.Core.load!(performance, [
          :count_of_reserved_seats,
          :count_of_unreserved_seats
        ])

      assert 0 = performance.count_of_reserved_seats
      assert 2 = performance.count_of_unreserved_seats

      BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)

      performance =
        BoxOffice.Core.load!(performance, [
          :count_of_reserved_seats,
          :count_of_unreserved_seats
        ])

      assert 1 = performance.count_of_reserved_seats
      assert 1 = performance.count_of_unreserved_seats
    end
  end
end

results = ExUnit.run()
```

## Play with the resources

```elixir
defmodule WhoTestsTheTesters do
  import BoxOfficeCoreTest.Helpers

  def run() do
    create_seating_chart()
    [seat] = create_seats().records
    populate_seating_chart()
    create_production()
    performance = create_performance()

    {:ok, _} =
      BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)

    # This match works (which is correct), but the same thing fails in the test
    {:error, _} =
      BoxOffice.Core.Reservation.make(create_patron().id, performance.id, seat.id)
  end
end

BoxOfficeCoreTest.Helpers.reset()
WhoTestsTheTesters.run()
```
4

Oh, interesting. Probably in the beginning of your Livebook or in setup you can do something like Application.put_env(:ash, :disable_async?, true)

5

I tried that and also tried using the configuration portion of Mix.install (both can be seen in the code above), but no dice.

But hey, the code works, it just doesn’t test well in Livebook for me, so I’ll move on and maybe come back and try to figure that out later. Those assert macros are nice, but I can just use regular matching for test purposes if I need to.

6

Yeah, that is super strange.

7

Thinking about it, I don’t think the disable_async bit really comes into play at all. That is standard for testing but in your case you aren’t using the sandbox adapter for the underlying repo, so it wouldn’t really matter.

I really can’t think of anything special that would cause this to behave differently in a test in a livebook :cold_face: