Need help debugging a weird @derive Any protocol issue that gives :nofile cannot_derive errors

I’m having a bit of an intermittent issue with a protocol with an Any implementation and a struct inheriting from that protocol and I’m not sure how to debug the issue or even reproduce consistently enough to file a bug so I thought I’d ask for suggestions here first.

I have a complex project with a couple protocols where this occurs. Unfortunately this project isn’t open source yet but hopefully soon. I was creating a textbook Any protocol to derive from similar to Protocols — Elixir v1.19.5 .

At first this bug arose when I stuck a dbg() statement into the Any implementation. When I did this and tried to derive the protocol I’d get something similar to:

== Compilation error in file lib/kerilixir/KERI/events/receipt_event.ex ==
\*\* (ArgumentError) could not load module Said.Saidifiable due to reason :nofile, cannot derive Said.Saidifiable for Kerilixir.Keri.ReceiptEvent
(elixir 1.19.5) lib/protocol.ex:331: Protocol.assert_protocol!/2
(elixir 1.19.5) lib/protocol.ex:1104: Protocol.derive/4
(stdlib 7.2) lists.erl:2641: :lists.foreach_1/2
(elixir 1.19.5) lib/protocol.ex:1097: Protocol.**derive**/3
lib/kerilixir/KERI/events/receipt_event.ex:22: (module)

When I removed the dbg() statement, removed the @derive, recompiled, added back the derive (without the dbg statement) the protocol would derive successfully. Adding the debug statement back would reproduce the error consistently.

Then somehow yesterday in fooling around with it I got the compilation and tests to pass with dbg() statement allowing me to troubleshoot the function call. Now when trying to derive other structs I’m getting this error consistently again. At this point, I don’t have the dbg() calls anymore in the Any implementation.

It seems like this is an issue in consolidation or compilation or something else that I don’t understand and so I’m reaching out for any ideas or even places to look in beginning to debug this. Its very frustrating because I’d just like to derive my implementation for most test cases and have a few exceptions with custom implementations but so far that has eluded me.

It does seem to happen more with mix test than mix compile if that changes things up. Then again, now its just occurring over and over with mix compile.

I tried to create a simpler example of the file structure and test structure I’m running here (tests fail but not the :nofile error) but failed to reproduce my error in this simpler example. GitHub - daidoji/simple_example

tl;dr something weird is happening in consolidation or compilation process and my protocol isn’t getting derived even though it should. Looking for ideas on how to troubleshoot as its intermittent and annoying.

Update: Running mix clean and then mix compile again gives a different struct whence this :nofile error appears. Removing the @derive from that particular module causes the rest of the modules to derive it successfully.

Hi @daidoji! Try compiling with `mix compile –profile time` and see if there are any messages related to compiler deadlocks. It may be that the module was indeed not yet compiled. This can happen if in your code or a dependency you are using try/rescue at compile-time to rescued undefined modules.

1 Like

There were two Finished deadlock resolution in 0ms messages in the output but nothing other than that. I don’t actually see the module file where the protocol is defined in this output.

...
[profile]     58ms compiling +      5ms waiting for module KeyManager.Backend while compiling lib/kerilixir/key_management/insecure_map_backend.ex
[profile]     19ms compiling +      0ms waiting while compiling lib/kerilixir_web.ex
[profile]     95ms compiling +      0ms waiting while compiling lib/kerilixir/repo.ex
[profile]     15ms compiling +      0ms waiting while compiling lib/kerilixir_web/controllers/error_json.ex
[profile]    171ms compiling +      0ms waiting while compiling lib/kerilixir/schemas.ex
[profile]    100ms compiling +      0ms waiting while compiling lib/kerilixir_web/controllers/page_controller.ex
[profile]     34ms compiling +      0ms waiting while compiling lib/kerilixir_web/telemetry.ex
[profile]    142ms compiling +      0ms waiting while compiling lib/kerilixir_web/gettext.ex
[profile]   1627ms compiling +     74ms waiting for module Kerilixir.Cesr.CesrElement while compiling lib/kerilixir/CESR/CountCode/CntCodeGeneratorV1.ex
[profile]                    |    125ms waiting for module Kerilixir.Cesr.Utility.Base64Indicies while compiling lib/kerilixir/CESR/CountCode/CntCodeGeneratorV1.ex
[profile]    161ms compiling +      0ms waiting while compiling lib/kerilixir_web/endpoint.ex
[profile]    155ms compiling +      0ms waiting while compiling lib/kerilixir_web/router.ex
[profile]   1550ms compiling +    185ms waiting for module Kerilixir.Cesr.CesrElement while compiling lib/kerilixir/CESR/Primitive/OneCharFixedPrimitiveGenerator.ex
[profile]                    |     96ms waiting for module Kerilixir.Cesr.Utility.Base64Indicies while compiling lib/kerilixir/CESR/Primitive/OneCharFixedPrimitiveGenerator.ex
[profile]   2803ms compiling +     35ms waiting for module Kerilixir.Cesr.CesrElement while compiling lib/kerilixir/CESR/CountCode/CntCodeGeneratorV2.ex
[profile]                    |    171ms waiting for module Kerilixir.Cesr.Utility.Base64Indicies while compiling lib/kerilixir/CESR/CountCode/CntCodeGeneratorV2.ex
[profile] Finished deadlock resolution in 0ms
[profile]    340ms compiling +   1326ms waiting for module Inspect.Phoenix.LiveView.JS while compiling lib/kerilixir_web/components/core_components.ex
[profile]     19ms compiling +   1604ms waiting for module KerilixirWeb.CoreComponents while compiling lib/kerilixir_web/controllers/error_html.ex
[profile]     32ms compiling +   1595ms waiting for module KerilixirWeb.CoreComponents while compiling lib/kerilixir_web/controllers/page_html.ex
[profile]     74ms compiling +   1608ms waiting for module KerilixirWeb.CoreComponents while compiling lib/kerilixir_web/components/layouts.ex
[profile]                    |      8ms waiting for module KerilixirWeb while compiling lib/kerilixir_web/components/layouts.ex
[profile]     95ms compiling +   1609ms waiting for module KerilixirWeb.CoreComponents while compiling lib/kerilixir_web/controllers/cesr_live.ex
[profile] Finished deadlock resolution in 0ms

== Compilation error in file lib/kerilixir/KERI/events/receipt_event.ex ==
** (ArgumentError) could not load module Said.Saidifiable due to reason :unavailable, cannot derive Said.Saidifiable for Kerilixir.Keri.ReceiptEvent
    (elixir 1.19.5) lib/protocol.ex:331: Protocol.assert_protocol!/2
    (elixir 1.19.5) lib/protocol.ex:1104: Protocol.derive/4
    (stdlib 7.2) lists.erl:2641: :lists.foreach_1/2
    (elixir 1.19.5) lib/protocol.ex:1097: Protocol.__derive__/3
    lib/kerilixir/KERI/events/receipt_event.ex:22: (module)

If the error happens exactly after the deadlock, then I’d say that’s indeed the root cause. Something is depending on `Said.Saidifiable` at compile time and `Said.Saidifiable` is depending on something else, causing it to deadlock. You can analyze the “waiting for module” messages for further hints.

Hmm I’ve looked and while I certainly think you’re probably right about the circular dependencies they’re difficult to suss out from that output and just eyeballing the code. Are there other tools or approaches I could take other than eyeballing the code and this profile output? For instance Kerilixir.Cesr.Utility.Base64Indicies that shows up a lot actually imports nothing other than stdlibs. The Kerilixir.Cesr.CesrElement is a protocol. The other “waiting for module” modules are

daidoji@vLEIDALaptop1:\~/vLEIDA/kerilixir$ grep “waiting for module” profile.log | sed ‘s/.*waiting for module (\[a-zA-Z0-9.\]+) .*/\\1/’ | sort -u
Inspect.Phoenix.LiveView.JS
Kerilixir.Cesr.CesrElement
Kerilixir.Cesr.CesrFieldMap.VersionString
Kerilixir.Cesr.CountCode.Generator.CntCodeV1Generator
Kerilixir.Cesr.CountCode.Generator.CntCodeV2Generator
Kerilixir.Cesr.Index.OneCharIndexedPrimitiveGenerator
Kerilixir.Cesr.Primitive.Generator.FourCharacterTagPrimitives
Kerilixir.Cesr.Utility.Base64Indicies
KerilixirWeb
KerilixirWeb.CoreComponents
KeyManager.Backend

Most of which don’t have circular dependencies as they’re just importing from libraries and aren’t used anywhere else with the exception of the Generator modules which are macros that generate a lot of stuff. They import Base64Indicies and have various spec types and maybe implementations of other protocols though.

None of these really have an issue if I remove the Any protocol implementation discussed above and start deriving it on various structs.

Actually we can close this and disregard. I yeet’ed my repo at Gemini Pro and it figured out the fix was to move the defprotocol and Any implementation into its own file fixing the circular dependency (before I had it in the same file that the functional module was residing in). I guess the dependency checker uses files instead of modules?

Anyways, this is fixed. Thanks for your help.

1 Like