Compilation error due to struct name from internal dependency

Questions / Help
1

Hello

I got a Elixir library that I install using internal dependencies, because the library is private with no access to Private hex.pm. deps() in mix.exs is configured with the current line, which has been working great (so far).

{:protocols, git: "git@<redacted>/protocols.git", tag: "0.1.5", sparse: "protocols-elixir"},

When I try to run the app using mix run or while build the Docker image, I’m facing the current error that stops the compilation:

** (CompileError) lib/brain/core/utils/to_protocols.ex:5: Protocols.TimeSeries.__struct__/1 is undefined, 
cannot expand struct Protocols.TimeSeries. 
Make sure the struct name is correct. 
If the struct name exists and is correct but it still cannot be found, you likely have cyclic module usage in your code
#24 3.797     lib/brain/core/utils/to_protocols.ex:4: (module)
------

defmodule Brain.Core.Utils.ToProtocols do
  def to_timeseries(forecast) do
    %Protocols.TimeSeries{points: to_points(forecast)}
  end
  [...]
end

Maybe I have a problem of import or a cyclic usage, but the weird behavior is that,

  • I got the compilation issue at 0.1.5
  • I’m updating the dependency down to 0.1.4 with mix deps.get + mix run, the app is compiling and running seamlessly.
  • I’m updating the dependency up to 0.1.5 with mix deps.get + mix run, the app is compiling and running seamlessly.
  • One code edition in the current project triggering a new compilation, and the issue with the library is back.

The Dockerfile (FROM elixir:1.13.0-slim) is facing the same issue, with those steps,

RUN mix local.rebar --force && mix local.hex --force

ARG MIX_ENV=prod

ENV LANG=C.UTF-8
ENV MIX_ENV=${MIX_ENV}

COPY mix.exs mix.lock ./
COPY config ./config
RUN mix do deps.get, deps.compile

COPY lib ./lib
RUN mix compile

COPY rel ./rel
RUN mix release

My current setup is,

elixir 1.13.0
erlang 24.1.7

Code can compile, which I conclude that there is no struct name issue or cyclic module usage, but somehow only when i’m {down,up}grading versions which is strange, leaving me without hypothesis left.

Do you have any ideas ? Or in need of further details ?

Thanks in advance !

2

where is your struct ‘Protocols.TimeSeries’ defined?
your module ‘Brain.Core.Utils.ToProtocols’ should know (alias or import) it

3

Protocols.TimeSeries is defined inside the :protocols library.

I have tried variations using alias or import with no success:

defmodule Brain.Core.Utils.ToProtocols do
  @moduledoc false

  alias Protocols.TimeSeries

  def to_timeseries(forecast) do
    %TimeSeries{points: to_points(forecast)}
  end
end

or

defmodule Brain.Core.Utils.ToProtocols do
  @moduledoc false

  alias Protocols

  def to_timeseries(forecast) do
    %Protocols.TimeSeries{points: to_points(forecast)}
  end
end

leads to the same error.

Meanwhile,

defmodule Brain.Core.Utils.ToProtocols do
  @moduledoc false

  import Protocols

  def to_timeseries(forecast) do
    %Protocols.TimeSeries{points: to_points(forecast)}
  end
end

leads to

== Compilation error in file lib/brain/core/utils/to_protocols.ex ==
** (CompileError) lib/brain/core/utils/to_protocols.ex:4: module Protocols is not loaded and could not be found
4

‘alias’ does not throw an error if the argument is missing, but ‘import’ does.
The problem is that it does not find your ‘Protocols’ parent module/directory.
How is the directory structure of the ‘lib’ folder where the ‘Protocols.TimeSeries’ module resides? You can list it with ‘tree lib/’ if the ‘tree’ tool is installed

5

Given that Protocols.TimeSeries is in another library, the file is located at deps/protocols/protocols-elixir/lib/primitives/time_series.pb.ex. Others folders in the :protocols lib have the same structure or file extension.

defmodule Protocols.TimeSeries do
  @moduledoc false
  use Protobuf, protoc_gen_elixir_version: "0.10.0", syntax: :proto3

  field :points, 1, repeated: true, type: Protocols.Point
end

When compiling, a corresponding _build/dev/lib/protocols/ebin/Elixir.Protocols.TimeSeries.beam file is built.

6

strange, I have no idea.
How does it behave with other modules from the ‘:protocols’ package?
Are there other modules like ‘TimeSeries’ that work fine?

maybe somebody else can help soving the issue?

7

seeing the file name /ebin/Elixir.Protocols.TimeSeries.beam …
is it possible that you have to do a
alias Elixir.Protocols.TimeSeries despite the module id declared with defmodule Protocols.TimeSeries ?
its just an idea.

8

Others modules in the lib have the same consistent faulty behavior. It depends on which module the app is trying to compile first.

Thanks for the help!

9

I tried adding Elixir. in front of aliases or imports with no successful effect.

10

Can you show the source tree of the library as it is in its own directory (and not how it is in your deps)?

11

The library tree looks like this

lib/
├── commands
│   ├── current_command.pb.ex
│   ├── power_command.pb.ex
│   └── setpoint_command.pb.ex
├── events
│   └── thermal_zone_updated.pb.ex
├── primitives
│   ├── point.pb.ex
│   └── time_series.pb.ex
├── services
│   └── predictive_control.pb.ex
├── signals
│   ├── constraint.pb.ex
│   ├── forecast.pb.ex
│   └── objective.pb.ex
└── state
    ├── storage.pb.ex
    └── zone.pb.ex

All pb.ex files are generated using protoc. The mix.exs looks like

defmodule Protocols.MixProject do
  use Mix.Project

  def project do
    [
      app: :protocols,
      version: "0.1.0",
      elixir: "~> 1.13",
      deps: deps(),
      package: package()
    ]
  end

  defp package do
    [
      organization: "world",
      files: [
        "lib/**/*.pb.ex",
        "mix.exs"
      ]
    ]
  end

  def application do
    [
      extra_applications: []
    ]
  end

  defp deps do
    [
      {:protobuf, "~> 0.8.0"}
    ]
  end
end
12

maybe it is a protoc internal but I wonder that the module Protocols.TimeSeries is inside the primitives folder.
Do the other modules also have Protocols as a parent module even if they are in different parent folders? For example:

/lib/commands/power_command.pb.ex ->  defmodule Protocols.PowerCommands
/lib/state/storage.pb.ex          ->  defmodule Protocols.Storage
13

Yes, all the modules are under the form Protocols.<Module>, like Protocols.PowerCommand, Protocols.Constraint or Protocols.Storage. Protocols as parent and only one child level.

I don’t understand how Protobuf can influence the compilation, as bumping versions down and up fixes the problem temporarily. Quite an investigation.

14

ok, I already suspected it … hmmm …
Does the problem occur if you delete the generated _build folder? If deleting the _build folder helps, then you may investigate under which condition the deletion is needed

15

Yes, rm -rf _build/ deps/ & mix deps.get & mix run --no-halt generates the same error. I found the same behavior while building the Dockerfile without using any cached layers.

Bumping once or twice the versions is the only way to make the app working. Editing one module where Protocols is needed, or used in another module using Protocols, and the compilation goes crazy.

16

as a last resort … can you talk to the creator of the :protocols library?

17

I can, that is me. Sorry if it hasn’t been clear ! :smile:

We are using this lib (in another repo) because it is generating “protocols” and packages from protbuf definitions usable for our Elixir and Python services that needs to speak together.

(Thaks for all the help.)

18

:joy:
anyway, I hope I could help a bit.

1 Like
19

If you fire iex -S mix in the directory of the library – not the app – can you then reference these modules e.g. with exports(Protocols.TimeSeries)?

1 Like
20

Looks like the thread is going to be very long without any progress. I would recommend to invest a bit of your time to create a minimal git or zip project. In many cases when doing so you can easily find an issue on yourself and if that would not be a case then you can share such sample project with community, so others may try to reproduce the problem and investigate it separately without your further responses.

The other option may be a temporary invite somebody to your private project. That would require less work from you, but obviously you would need to trust that person. I’m senior developer with many bug reports submitted from time to time in elixir, ecto, phoenix and few other projects, for example Mix test fails in umbrella (Ecto, Phoenix): db couldn't be dropped

If interested here’s my profile: Eiji7 (Tomasz Marek Sulima) · GitHub

1 Like