Dialyzer complaint about MapSet.member not getting proper type as argument, possible specs bug in MapSet

Questions / Help
1

I got this module

defmodule MapSetBug do
  @vars [
    %{one: 1, two: MapSet.new(["a", "b", "c"])},
    %{one: 2, two: MapSet.new(["a", "b", "c"])},
    %{one: 3, two: MapSet.new(["a", "b", "c"])}
  ]

  @map_set Enum.flat_map(@vars, fn var ->
             Enum.map(var.two, &{var.one, &1})
           end)
           |> MapSet.new()

  def valid(one, two) do
    MapSet.member?(@map_set, {one, two})
  end
end

Which gives me this error:

lib/map_set_bug.ex:13:no_return
Function valid/2 has no local return.
________________________________________________________________________________
lib/map_set_bug.ex:14:call_without_opaque
The call MapSet.member?(#{'__struct__':='Elixir.MapSet', 'map':=#{{1 | 2 | 3,<<_:8>>}=>[]}, 'version':=2},{_,_}) does not have an opaque term of type 'Elixir.MapSet':t(_) in 1st.
________________________________________________________________________________
done (warnings were emitted)

I am unable to resolve it. But this:

defmodule MapSetBug do
  @map_set MapSet.new()

  def valid(x) do
    MapSet.member?(@map_set, x)
  end
end

Gives the same kind of error:

lib/map_set_bug.ex:4:no_return
Function valid/1 has no local return.
________________________________________________________________________________
lib/map_set_bug.ex:5:call_without_opaque
The call MapSet.member?(#{'__struct__'=>'Elixir.MapSet', 'map'=>#{}, 'version'=>2},_x@1::any()) does not have an opaque term of type 'Elixir.MapSet':t(_) in 1st.
________________________________________________________________________________

Now
 is it a dialyzer bug? MapSet specs bug? Or my code bug?

Of course this:

defmodule MapSetBug do
  def map_set do
    MapSet.new()
  end

  def valid(x) do
    MapSet.member?(map_set(), x)
  end
end

Gives no dialyzer error, but that’s not something I want to achieve. I need that MapSet created once at compile time, and not every time a function is called.

4 Likes
2

As a MapSet.t is opaque, you are not allowed to use it literally outside of the MapSet module. But you are doing exactly that by creating one at compiletime and putting it into a module attribute.

1 Like
3

So what is the solution? I’m not allowed to use MapSet as module variable, even though it’s perfectly valid code, and works as intended?

4

Create it at runtime.

5

I’m sorry I’m not following. Can you elaborate more? I have a feeling that you propose a wrong workaround instead of a proper solution. The problem is with specs and not the code itself. Changing implementation because there is a bug in specs, and not the implementation of the code itself, does not seem to be a step in right direction.

1 Like
6

There is no bug in the spec. MapSet.t is an opaque type.

@foo MapSet.new() will be evaluated at compile time to %MapSet{}, and then in your module all occurences of @foo will be replaced with this literally, and you are not allowed to literally use %MapSet{} or any of its forms because its an opaque type.

The only alternative is to fully expose the type, but that again would mean, that all future versions would be tied the concrete internal representation of the type, and therefore lead to problems improving the MapSet algorithms.

3 Likes
7

I understand that %MapSet{} is an internall type, a private Elixir one if you may, and you explicitly should not use it.

Still, the code I presented is working without any problems. I’m not using any internal API’s calls. And yet it gives me dialyzer errors, while this:

defmodule MapSetBug do
  @map_set MapSet.new()

  def valid(x) do
    Map.has_key?(@map_set, x)
  end
end

Gives not, even though it’s not so different in implementation. And if we go deeper, and just explicitly copy implementation of MapSet.member/2 into our function like this:

defmodule MapSetBug do
  @map_set MapSet.new()

  def valid(x) do
    match?(%{^x => _}, @map_set)
  end
end

Still, no error. So even if the specs are valid for the MapSet module in Elixir, the design of those seems to be flawed, and yet you try to convince me otherwise.

8

Again, the type is @opaque, you are not allowed to use it in a way that produces a immediate value that you pass in a function that expects a MapSet.t, dialyzer will complain under exactly this conditions.

9

You are explaining to me why the error happens, but I understand why it happens.

What I am trying to say, that because a MapSet can be created on compile time (and why shouldn’t it), the current implementation of specs is not good enough. I don’t have better specs at the moment. But I’m not sure if you disagree with me, and if so, I still don’t understand your arguments.

You seem to (but I may be wrong in that assumption) trying to tell me, that using MapSet module functions on MapSet created at compile time is something I should not do, because specs were written that way. But if we just for the sake of argument forget about the specs at all. Is there any other valid reason why shouldn’t use MapSet module functions on MapSet created at compile time? And if not, doesn’t that indicate problem with dialyzer specs for MapSet module?

2 Likes
10

Everything could land in your code as an immediate value because of compiletime evaluation. To get rid of this you can either not use dialyzer or not using the immediate values of an opaque type.

Changing the types to public will cause much more problems than leaving it opaque.

11

With all said, dialyzer is not type inference, it is success typing, if you think that you are not messing up and still it complains to you, you can disable it on for that expression

1 Like
12

I think that your scenario is perfectly valid, and I agree that it shouldn’t lead to a dialyzer error, but unfortunately you’re hitting the limitation of the dialyzer, MapSet typespec, and the fact that you’re generating the data during compilation.

A potential solution here would be to remove MapSet opaqueness in Elixir (wdyt @josevalim?). Even if that’s done, you’ll need to wait for the next Elixir version, and even then it’s only going to solve the problem for MapSet, not for other opaque types.

So in the meantime, you can reach for a bit of ugly trickery to confuse the dialyzer:

@doc false
@spec map_set :: MapSet.t
def  map_set(), do: apply(__MODULE__, :map_set_intern, [])

@doc false
def map_set_intern(), do: @map_set

And then only use map_set() to work with the mapset. I agree it’s ugly, but it should do the job (haven’t verified it though). If you take this path, I also advise adding a comment to properly explain this trickery.

3 Likes
13

Alternative is to use @dialyzer module option to ignore warnings in the given function (docs can be found there).

3 Likes
14

What about the chance of a different version for compilation vs the version used at runtime? I mean in practise this would probably lead to lots of other issues as well and it‘s unlikly to happen, but in this case it could mean the module attribute could use a different internal implementation than what is expected by the module at runtime.

15

I’ve created a github issue on the elixir repo about this 3 months ago: https://github.com/elixir-lang/elixir/issues/8463

tl;dr: It’s not a bug in your code, it’s just dialyzer not being very flexible. Eventually somebody will write a patch for this very legitimate use case, but for the time being you can safely ignore the warning.

2 Likes
16

Thank you all for commenting. And @1player thank you so much for linking the issue. At least I know that the core team is aware of the problem.

2 Likes
17

In case this is useful for anyone in the future: I handled this by disabling the warning for the specific functions that contained the “problematic” code:

  @allow_list MapSet.new(["a", "b"])

  @dialyzer {:no_opaque, allow_listed?: 1}
  defp allow_listed?(identity) do
    MapSet.member?(@allow_list,  identity)
  end

PS I used Erlang -- dialyzer for hints re warning names.

2 Likes
18

[deleted]

19

It is not a dialyzer bug.

It is a missuse of immediate values where you are not allowed to use them.

By no means you are allowed to create or match a value of any @opaque type outside of the module that defines it.

Dialyzer has no notion of “this immediate value has been created during compile time using only the allowed functions”. Dialyzer only sees, there is an immediate value in the modules bytecode.

If you want to surpress such a message us your own Macro that marks the code as generated: true, which will surpress some, but not all dialyzer complaints.

1 Like
20

You are generally allowed to create opaquely typed values outside of their module (using helper functions), that’s the whole point of opaque types – and that is what is happening here – in the original post – yes, in the later posts there are violations of that by matching.