Tests suddenly not working anymore

Questions / Help
,
1

Hi all,

I think I’ll feel silly when this problem will be pinned down.

The tests on my application are suddenly not working anymore. Same commit, same CI run. I can’t find what changed.

The actual error:

 ** (FunctionClauseError) no function clause matching in Access.get/3

This comes from my test file, actually, I want to invoke the get/3 macro from WsdataselectWeb.ConnCase. So nothing fancy, just the classic Phoenix way to test a GET request.

It is like if Elixir lost track of the Wsdataselect.ConnCase module and maps the get instruction to the Access module.

Anyone can help me understand what’s going wrong ?

This is one test file where the problem occurs: Blame · test/permissions_test.exs · phnx · OSUG / RESIF / wsdataselect · GitLab

defmodule PermissionsTest do
    use WsdataselectWeb.ConnCase, async: false

    test "User tries to read forbidden network", %{conn: conn} do
      conn =
        conn
        |> get(~p"/query?net=FR&sta=CIEL")

      assert conn.assigns.forbidden_networks == ["FR"]
      assert conn.assigns.datafiles == []
    end
end

Full output of the test:

  1. test Permissions test User tries to read forbidden network (PermissionsTest)
    test/permissions_test.exs:166
    ** (FunctionClauseError) no function clause matching in Access.get/3

    The following arguments were given to Access.get/3:

    # 1
:ok

# 2
"countryCode"

# 3
nil

    Attempted function clauses (showing 6 out of 6):

    def get(%module{} = container, key, default)
def get(map, key, default) when is_map(map)
def get(list, key, default) when is_list(list) and is_atom(key)
def get(list, key, _default) when is_list(list) and is_integer(key)
def get(list, key, _default) when is_list(list)
def get(nil, _key, default)

    code: |> get(~p"/query?net=FR&sta=CIEL")
    stacktrace:
    (elixir 1.19.4) lib/access.ex:324: Access.get/3
    (wsdataselect 0.9.0) lib/delivery_metrics/delivery_metrics.ex:28: DeliveryMetrics.compute/1
    (wsdataselect 0.9.0) lib/wsdataselect_web/controllers/query_controller.ex:1: WsdataselectWeb.QueryController.action/2
    (wsdataselect 0.9.0) lib/wsdataselect_web/controllers/query_controller.ex:1: WsdataselectWeb.QueryController.phoenix_controller_pipeline/2
    (phoenix 1.8.1) lib/phoenix/router.ex:416: Phoenix.Router.call/5
    (wsdataselect 0.9.0) lib/wsdataselect_web/endpoint.ex:1: WsdataselectWeb.Endpoint.plug_builder_call/2
    (wsdataselect 0.9.0) lib/wsdataselect_web/endpoint.ex:1: WsdataselectWeb.Endpoint.call/2
    (phoenix 1.8.1) lib/phoenix/test/conn_test.ex:225: Phoenix.ConnTest.dispatch/5
    test/permissions_test.exs:171: (test)

And there is the successful test CI run (6d ago) : test (#861557) · Jobs · OSUG / RESIF / wsdataselect · GitLab

And here the failing test CI run: test (#865942) · Jobs · OSUG / RESIF / wsdataselect · GitLab

I tried to change the elixir/otp versions but have the same result.

I must also add that I can reproduce the problem locally on my laptop.

2

Looking at the branch history, you have updated Erlang and Elixir before this commit. And also the Debian Docker image version. Does temporarily returning Erlang & Elixir versions back to their older versions help?

It looks like you could try to manually play with this and fix it by calling DeliveryMetrics.IPWho functions in an iex session.

2 Likes
3

Hi, thanks a lot for your reply !

You nailed something by mentionning my DeliveryMetrics.IPWho module

It relies on an external service and they just changed their URL ( from ipwho.org to api.ipwho.org). Fixing the URL also fixes the problem.

Actually my tests spotted a real problem, thats so nice :slight_smile:

I capture the exception in my code, I need to dig a bit and understand where this get happens to make the error message clearer and manage the problem gracefully.

1 Like
4

You probably also don’t want to actually have your unit tests making real requests that hit their API. You’ll want to look into mox | Hex or bypass | Hex .

1 Like
5

I like Mimic and Patch more but yes, fair point on not doing real requests in tests.

6

Ironically it seems like this thread is a pretty good case in point in favor of hitting real APIs in tests. Using mocks here just means catching bugs in production.

Best to separate those tests out (tag them) and invoke them sparingly, though, just to be respectful.

3 Likes
7

To go one step further you also want to be testing the structure of what you get back in order to ensure the response (JSON, XML, etc) has changed. Never test the content that is returned, just do a schema check. Once again, separate this out and do it sparingly, but better to have a test fail because of a JSON change then for production to blow up.

2 Likes
8

Absolutely, I would prefer to always do that but most (read: 99.9%) APIs out there don’t have sandbox environments. Nobody wants to be hit with +2500% API usage invoice because one CI worker looped and ran 300 times.

1 Like
9

Warm thanks for all your comments.

I’m lucky not to have mocked the API, but I think both approaches are complementary. I’ll add the mox test and use tags to select them easily.
One real API test is good, and deeper unit tests using mox seems fair.