Elixir 1.19 upgrade issues - compiler warnings + test failures

Questions / Help
1

Hello,

I’m trying to upgrade a pretty large project from Elixir 1.18.4 to Elixir 1.19.1 (1321 modules), but seeing some compiler warnings saying modules aren’t available.
These show with both OTP 27 and OTP 28, so I’m guessing it’s related to Elixir 1.19 changes rather than OTP.

Erlang/OTP 28 [erts-16.1] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit:ns]

    warning: you are implementing a protocol for Foobar.Resources.User.V2.Api but said module is not available. Make sure the module name is correct. If Foobar.Resources.User.V2.Api is an optional dependency, please wrap the protocol implementation in a Code.ensure_loaded?(Foobar.Resources.User.V2.Api) check
    │
 43 │ defimpl Bamboo.Formatter, for: Foobar.Resources.User.V2.Api do
    │ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    │
    └─ lib/foobar/email/recipient.ex:43: (file)

    warning: you are implementing a protocol for Foobar.Resources.Baz.V1.Api but said module is not available. Make sure the module name is correct. If Foobar.Resources.Baz.V1.Api is an optional dependency, please wrap the protocol implementation in a Code.ensure_loaded?(Foobar.Resources.Baz.V1.Api) check
    │
 49 │ defimpl FunWithFlags.Group, for: Foobar.Resources.Baz.V1.Api do
    │ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    │
    └─ lib/foobar/feature_flags/group.ex:49: (file)

The modules exist, so I’m not sure why defimpl is throwing warnings or why only these two, we have dozens more that look near identical that don’t.

We’re also seeing extra warnings when running tests, from some modules only compiled for tests, through elixirc_paths config like this:

defp elixirc_paths(:test), do: elixirc_paths(:dev) ++ ["test/support", "test/resources", "test/benchmarks"]

defp elixirc_paths(_), do: ["lib", "web"]

warning: invalid association `test_comment_test_categories` in schema Foobar.Test.Support.Framework.Resources.TestCategory.V1.Api: associated schema Foobar.Test.Support.Framework.Resources.TestCategory.V1.JoinTables.TestCommentTestCategory does not have field `api_id`
└─ test/support/framework/resources/test_category/v1/api.ex: Foobar.Test.Support.Framework.Resources.TestCategory.V1.Api (module)

warning: invalid association `test_editor_test_posts` in schema Foobar.Test.Support.Framework.Resources.TestPost.V1.Api: associated schema Foobar.Test.Support.Framework.Resources.TestPost.V1.JoinTables.TestEditorTestPost does not have field `api_id`
└─ test/support/framework/resources/test_post/v1/api.ex: Foobar.Test.Support.Framework.Resources.TestPost.V1.Api (module)

warning: invalid association `test_editor_test_posts` in schema Foobar.Test.Support.Framework.Resources.TestEditor.V1.Api: associated schema Foobar.Test.Support.Framework.Resources.TestEditor.V1.JoinTables.TestEditorTestPost does not have field `api_id`
└─ test/support/framework/resources/test_editor/v1/api.ex: Foobar.Test.Support.Framework.Resources.TestEditor.V1.Api (module)

warning: invalid association `test_comment_test_categories` in schema Foobar.Test.Support.Framework.Resources.TestComment.V1.Api: associated schema Foobar.Test.Support.Framework.Resources.TestComment.V1.JoinTables.TestCommentTestCategory does not have field `api_id`
└─ test/support/framework/resources/test_comment/v1/api.ex: Foobar.Test.Support.Framework.Resources.TestComment.V1.Api (module)

warning: invalid association `test_post_test_categories` in schema Foobar.Test.Support.Framework.Resources.TestPost.V1.Api: associated schema Foobar.Test.Support.Framework.Resources.TestPost.V1.JoinTables.TestPostTestCategory does not have field `api_id`
└─ test/support/framework/resources/test_post/v1/api.ex: Foobar.Test.Support.Framework.Resources.TestPost.V1.Api (module)

warning: invalid association `test_post_test_categories` in schema Foobar.Test.Support.Framework.Resources.TestCategory.V1.Api: associated schema Foobar.Test.Support.Framework.Resources.TestCategory.V1.JoinTables.TestPostTestCategory does not have field `api_id`
└─ test/support/framework/resources/test_category/v1/api.ex: Foobar.Test.Support.Framework.Resources.TestCategory.V1.Api (module)

These are for our internal framework tests and are causing over 100 test failures.
Everything compiled with no warnings with Elixir 1.18.4, and no test errors.

Our internal framework is pretty macro heavy, so maybe some behavior changed there?
All those JoinTables modules are dynamically created by our framework through macros, but only the test/support ones are failing.
If it was totally broken, I’d expect to see test failures or warnings for our actual application rather than only this framework code.
Not really sure where to start looking. Any ideas?

Thanks

2

Do you use Ash? If so, upgrade to latest Spark. The Ecto ones will be solved by upgrading to a more recent Ecto too.

In any case, all of those issues are about Elixir failing to find modules at compile-time but they are still available at runtime, which is why everything works. I assume once you upgrade your deps most of them will disappear.

3

We’re starting to use Ash, but these warnings are from code elsewhere in our app. They’re part of an in-house framework we’ve built a while back that also uses macros to wire things up.

All of our dependencies are on the latest published versions except for:

  • ash_jason
  • goth
  • stripity_stripe

Don’t think they’d be the issue.

which is why everything works.

Yeah, I think everything might work other than the test failures. That’s the main thing I’m looking to fix, I can probably resolve the defimpl warnings by tossing a Code.ensure_compiled! or two in there. Not sure why it can’t find them at compile-time, but could be some macro weirdness.

I’m guessing the test failures I’m getting are related to the invalid association warnings, though. All the failing tests are related to those associations, like these:

(ArgumentError) unknown field `:api_id` in %Foobar.Test.Support.Framework.Resources.TestComment.V1.JoinTables.TestCommentTestCategory{__meta__: #Ecto.Schema.Metadata<:built, "test_comment_test_categories">, test_category_id: nil, test_comment_id: nil, position: nil, inserted_at: nil, updated_at: nil}


     Expected Status: 200
     Actual Status: 500
     Errors: [
       {
         "detail": null,
         "code": "Elixir.Ecto.QueryError",
         "detail": "deps/ecto/lib/ecto/association.ex:871: field `api_id` in `where` does not exist in schema Foobar.Test.Support.Framework.Resources.TestComment.V1.JoinTables.TestCommentTestCategory in query:\n\nfrom t0 in Foobar.Test.Support.Framework.Resources.TestComment.V1.JoinTables.TestCommentTestCategory,\n  join: a1 in Foobar.Test.Support.Framework.Resources.TestCategory.V1.Api,\n  on: is_nil(a1.deleted_at) and t0.nil == a1.id,\n  where: t0.api_id == ^\"d0a792bb-8665-4d0c-bc2f-afc9a6bcfef5\",\n  order_by: [asc: t0.api_id],\n  select: {t0.api_id, t0}\n",
         "id": "GHD2Od1x_ijf-moACYpI",
         "meta": {},
         "stacktrace": [
           "(elixir 1.19.1) lib/enum.ex:2520: Enum.\"-reduce/3-lists^foldl/2-0-\"/3",
           "(elixir 1.19.1) lib/enum.ex:1814: Enum.\"-map_reduce/3-lists^mapfoldl/2-0-\"/3",
           "(elixir 1.19.1) lib/enum.ex:2520: Enum.\"-reduce/3-lists^foldl/2-0-\"/3",
           "(ecto 3.13.3) lib/ecto/repo/queryable.ex:223: Ecto.Repo.Queryable.execute/4",
           "(ecto 3.13.3) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3",
           "(elixir 1.19.1) lib/enum.ex:1688: Enum.\"-map/2-lists^map/1-1-\"/2",
           ""
         ],
         "status": 500,
         "title": null
       }
     ]
4

Don’t think they’d be the issue.

The reason I asked is because there was a bug related to Ash and dependencies which caused exceptions similar to yours, and a new version was published yesterday. :slight_smile:

The unknown fields are are separate issue though. If you are defining fields programatically, depending if a module exists or not, then that would be the root cause. If either Ash or your framework use Code.ensure_compiled! more consistently (instead of just checking for it), you should have reliable compilation errors. The reason we can’t use Code.ensure_compiled! when checking for protocols is backwards compatibility.

1 Like
5

Thanks! I’ll poke around areas like that in our framework and see if I can get things working.

Update: Found a spot where we were creating a module inside of a macro, added a call to Code.ensure_compiled!/1 after that and seems to have fixed all the problems. :slight_smile:

1 Like