Proposal: Private modules (general discussion)

Elixir News & Announcements Elixir News
141

Hm… Interesting, I didn’t think of it, but it’s a pretty good idea.

In this case, expose would be great! you call expose to expose your private module and then do whatever you want with it. You can require it, import it or use it :slight_smile:

I think you should either add export and nothing else OR add all four aliasp, requirep, importp and usep. Both are interesting extremes. The former adds a new concept, which is to expose a normally hidden module. The you can do whatever you want with that module. The latter adds a more succint API to work with private modules as if they were public.

After your remark I prefer expose.

EDIT: and you could probably perform all kinds of async deferred error checking on expose if you want, as you were proposing with aliasp

2 Likes
142

I feel very much like @gregvaughn (what a great analogy with puppies!), and would also prefer proposal A.

Elixir is very compact and simple compared to OO languages I’ve used in the past, and I love the direction it’s taken, with changes happening more and more in the ecosystem rather than the language. I’m not too excited about the fancier proposals which mean more cognitive load, especially for new users, and may make the debugging more difficult, especially when we’ve reached a stage where the language is supposed to be quite stable.

I would compare this to Dialyzer’s opaque types. They’re best effort warnings, but still provide practical utility, without adding hard walls to the language. I recall playing with OTP libraries like arrays and queue and being able to play with their data structure and easily understand how they’re implemented. It felt amazing to create a queue in the console and see {[], []}, plain and obvious, rather than some kind of <Queue object at 0x2aba1c0cf890>.

Another minor thing I like about proposal A is there’s no repetition, the fact that a module is private or public is visible in that module but doesn’t affect the callers (no require+alias, aliasp, etc.)

1 Like
143

Hi everyone, thanks for the feedback so far.

If we assume that the issue is inadvertent use of private APIs, then proposal A is the simplest solution that solves the problem.

While I mentioned earlier that warnings are not enough, they are not enough only if someone wants to bypass or ignore those warnings. However, if someone is in a position where they don’t care about warnings or where they want to bypass the visibility system, none of the proposals permanently address it.

Also, given that this is not an issue equally shared by all members of the Elixir community, but it is an important one for some projects, including Elixir itself, it makes sense to pick the solution with the smallest footprint (in terms of implementation, complexity, concepts, etc) possible, to avoid increasing the cognitive load for those not using it, which again, is solution A.

While I personally think the alias system in B/C/D provide more flexible use cases than A, such alias system will have a bigger footprint in the language either in the API (i.e. by introducing aliasp) or in the usage (i.e. requiring multiple require+alias calls).

In other words, I am personally stating my preference at the end of the discussion goes towards option A . Thanks @hauleth for initially proposing it.

Note we may not implement proposal A immediately, as it is wise to introspect about this a bit further and wait a bit more to see if other solutions to the problem surface. But if we do implement it, it will be closer to A in nature (probably preceded by a discussion on the API).

Have a good weekend!

17 Likes
144

Hi Jose. I’m a bit late to the party, but, how about using defmodulep to define a file-private module? I.e., a module that’s only visible within the file itself. To other modules in the file, it exists. To modules in other files, it doesn’t exist. Semantically, it’s similar to defp–a private function is visible only to other functions in the same module, and invisible to functions in other modules. So where defp applies visibility within a module, defmodulep applies visibility within a file.

1 Like
145

It’s not going to work. This proposal is for something else. Look that as in Elixir core as in Elixir libraries there are lots of modules which are internal. They are required by other modules in library/project, but they should not be used outside it. Unfortunately at compiler level it’s not possible to determine in which app you are, so we can’t reserve specific module only for current app. Previously developers were just defined @moduledoc false, but lots of other developers (even those which maintain popular libraries) were still using API which was not documented. Such internal API could be changed basically at any commit. We are discussing here a way to not allow such cases.

From this module which is visible only in one file is not really usable. While it could be used it would cause creating a big file with even thousands of lines which contains probably most of modules. Here we want to keep already defined code, but (for example) just change some definitions i.e. defmodule MyLib.MyModule do … end -> defmodulep MyLib.MyModule, visible_to: [MyLib] do … end. Similarly it goes for other proposals where (again for example) we want to add only one extra module attribute.

146

I see, thanks for the context–requirement is minimal changes to existing libs. Gotcha.

147

I wonder if it’d help to look at a few of the concrete examples of private APIs that have caused problems during Elixir upgrades?

1 Like
148

For example absinthe have a problem with changing Elixir version to 1.7.x if I remember correctly. Also there was similar problem with amnesia library. Basically here is typical internal API example:

defmodule MyModule do
  @moduledoc false

  # …
  # nothing from here should be called outside current app
  # but other modules in current app should be able to call everything
end
149

I’m wondering more about the actual situations, not just an example.

As in, did people know they were reaching into private modules and just really “needed” the function? Or was it accidental, they just didn’t notice the @moduledoc false annotation in a thousand line file?

Has it always been an attempt to access a private module function, or was it wound up in the details of how something more complex worked, like a macro doing something wild, or the behavior of some kind of module attribute…

1 Like
150

@moduledoc false is used mostly at start of module. I did not saw it in middle or at bottom, but of course it does not mean nobody did that. Anyway @moduledoc as well as @doc and @typedoc allows to generate documentation which is available at hexdocs.pm. Using everything else than those modules/functions which are placed in this site is seen as bad and unsafe practice, because such internal API could be changed basically in any commit. If you see module.function/arity in docs this means it will stay and works same until new major version would be released. From this it’s just not possible to miss @moduledoc false as developers should not even take a look at internal modules in order to use them in their application.

It was always about using module.function/arity which documentation is not published.

# no documentation published
# don't use whole module
defmodule Example1 do
  @moduledoc false

  # …
end

# no documentation published for Example2.sample/0
# don't use only this function
defmodule Example2 do
  @doc false
  def sample, do: …

  # …
end
151

Here’s a few examples:

I think that you may be overestimating the number of people that will recognize a @moduledoc false attribute means that the module is private to the application. Also don’t forget that it is easy for a developer to search on Google, find some example code (that may be using a private module), and then test the code locally and everything works just fine without any warnings. There’s also at least 2 more blog posts that used Mix.Ecto when it was private, but I couldn’t find them again with a quick search.

2 Likes
152

The problem with @moduledoc false is, that one not only needs to see it, but also needs to understand it.

Does the one who calls into that module even know the difference (which only is by convention) between no @moduledoc/@doc at all and @moduledoc false/@doc false?

2 Likes
153

I’ve a few other examples as well:

Afaik @kip was aware he was using private API, but the issue is that there’s no proper incentive to speak to the maintainer to make it public vs. just using the private api as is – at least not yet. Private API in elixir (e.g. mix) is especially problematic because of the rather fixed release cycle. The goal of this proposal is to my understanding to force (or at least promote) the work/discussion of such cases to happen at an earlier stage – at the place where someone want’s to use a private api opposed to when the dependency (elixir, plug, ecto, …) finally updates and breaks things.

Even if the private API could just be made public a few month down the line a package could at least depend on the exact version, where it’s still private, call the private API and make a note to release a new version using any updated public API when it’s released. Anybody updating before the new release would get a warning that the package does not work with e.g. elixir 1.X yet. Not perfect, as the private API could still break before the next release, but at least both parties are aware of the issues and when it’s to be properly resolved.

4 Likes
154

It looks like for the absinthe case, it was more complex than just using a private API

1 Like
155

We work on a biggish Elixir app and this is something we spent few hours discussing, I believe that private modules is a must have and lean towards option D and C.

I don’t mind would be that error or a warning as we are able to return error on warnings (and we do it already), but with option D debugging would be harder, because you need to know that your function is magically lives in another module :"Elixirp.Foo.Bar".

Also, are private modules going to be accessible from tests unlike private functions? A private module can contain a lot of functionality and testing all edge cases in a high-level function that calls a private module would lead to code duplication and increased complexity. So I beleive we should test it as any other module.

One other thing I’m not sure is listing each caller module individually, namespacing looks more practical here - I would like to allow all modules in current domain context to call private module. But I’m not sure it’s possible to implement it this way because we don’t know all the modules within a context beforehand.

2 Likes
156

@sasajuric How does this mesh with what you said in your book that module name nesting is just syntactic sugar and doesn’t imply any special relationship between modules.

This proposal seems like it will trigger special relations between nested modules. Curious what your thoughts are.

157

The . character is just another character in the module name, so the runtime doesn’t care about it at all. Of course, the tooling can introduce additional rules and conventions (if I’m not mistaken, this is already done with protocols).

It doesn’t look like the proposal A introduces any special relationship based on the . character (or any other character), but it does introduce a special relationship based on the module attribute (@module_visible_to, or however it’s going to be called).

I personally like this proposal, since I was never really convinced that @moduledoc false is sufficient. This proposal looks lightweight, and it should help with a better communication of intent and make things more enforceable.

5 Likes
158

10 posts were split to a new topic: Alternative names to defmodulep

159

Folks, I will go ahead and close the thread, as it has reached its natural course. Of course, new discussions around the topic are welcome and can link back to it. Thanks!

5 Likes