Proposal: Private modules (general discussion)

moderators note:

A conclusion by @josevalim has been drawn in Proposal: Private modules (general discussion) - #143 by josevalim

While Elixir has private functions, it does not have the concepts of private modules. This makes it harder for applications and libraries to define clear boundaries and communicate intent clearly.

As an example, when Elixir v1.7 was released, it broke some libraries that were using Elixir’s private APIs. This gives an impression of instability and immutarity in the ecosystem. Even more worrying, is that this practice in the long term can be really harmful as systems grow in size. If we, as a community, fail to define boundaries and fail to respect compatibility, updating only a small part of the system becomes impossible, because a minimal change breaks many unwarranted things along the way. It usually goes like this: let’s update Elixir! Unfortunately, updating Elixir breaks package X because X used a private API. So we have to update package X too but wait! That breaks Y and Z. Soon you find yourself having to update the whole system at once.

For these reasons, it is desirable to have a better way to outline boundaries and communicate intent. This is not only useful when working with dependencies. Even within the same library or application, developers can use well-defined boundaries to better organize their codebase and reveal intent to their coworkers.

However, one of the questions on this topic is how strict does the private module system has to be as there are many situations we would like to bypass it.

As @scarfacedeb mentioned in another thread:

I can think of at least 2 uses of open private modules:

1. As @JEG2 said, sometimes you have to use private modules in IEx in prod. You may encounter an unexpected error that you didn’t anticipate in your code and the quickest way to debug it is to call internal modules by hand and check the results. You could also use tracing in these cases, but I don’t see why we can’t have both.

2. When I’m learning how a new library (or app) works, I often call its internal modules directly to experiment and get a better idea how they work under the hood. Now it’s easy to do in iex and it doesn’t require to now about any new concepts (such as private modules, their visibility, etc).

Making code easy to explore is useful in production and while learning too.

With this in mind, this proposal is going to highlight four possible implementations for further discussion. Before we get to the possible implementation, we need to establish some common ground. Note the APIs in this proposal are not final and are meant to be examples. Once an approach is chosen, we can have a separate discussion to refine its APIs.

Best-effort warnings

One possible implementation of “private modules” is to provide best-effort warnings. This would work by annotating the visibility of a module, such as:

defmodule MyApp.Private do
  @module_visible_to [MyApp]
end

Now invoking MyApp.Private outside of MyApp will emit a warning that the module is private and may not be accessible externally.

It is important to note that, when code is compiled, Elixir does not actually guarantee the module you are calling exist. For example, if you have this function:

defmodule Foo do
  def bar, do: Bar.baz
end

The code will compile even if Bar is not defined. While mix does warn in cases like this, those warnings are “best-effort”. For example, the code below, while semantically the same to the code above, won’t warn:

defmodule Foo do
  def bar do
    mod = Bar
    mod.baz
  end
end

which means that “best-effort warnings” for private modules can be easily bypassed by doing:

defmodule Foo do
  def bar do
    mod = MyApp.Private
    mod.baz
  end
end

Therefore, the only way we could consistently and constantly warning when breaking a private module boundary, is if the private modules are required (via require/2) before they are used. Otherwise, we can only provide best-effort warnings, which are extremely easy to bypass and may not display as frequently.

Guaranteed warnings/errors (defmodulep)

If we want to have guaranteed warnings, private modules must be explicitly required before usage. One possible implementation of such mechanisms is to introduce a defmodulep construct, that defines a module in a separate namespace:

defmodulep MyApp.Private, visible_to: [MyApp] do
  def hello do
    IO.puts "hello world"
  end
end

In the definition above, only MyApp and modules nested under it can access MyApp.Private. To access a private module, you must explicitly require and alias it:

defmodule MyApp.Other do
  require MyApp.Private, as: Private
  Private.hello
end

The require is necessary to validate the visibility rules. The alias is required to bring the private module to the current namespace. The require+alias mechanism is essential to this alternative.

If we decide to go on the defmodulep route, we have three options:

1. Modules must be explicitly required+aliased and it will error if you break its boundaries. The namespace the module will be assigned to is private, which means you have no official ways of accessing a private module beyond its original intent.

2. Modules must be explicitly required+aliased and it will error if you break its boundaries. However, the namespace the module will be assigned to is public, which means you can access it directly, without any visibility check, by using its long name. For example, defmodulep Foo.Bar would be accessible directly via :"Elixirp.Foo.Bar", which could also be stored in a variable and passed around.

3. Modules must be explicitly required+aliased but it warns instead of erroring if you break its boundaries.

Rejected ideas

The following ideas were rejected:

• Declaring the module visibility per package or application. The Elixir language and the compiler do not have the concept of “applications”. Applications and packages are purely a build tool construct. In a way this is great, because the language is small and we build features on top, but it also means we cannot implement a construct such as visibility per package as part of the language.

Proposals

With this in mind, we have four proposals (A, B, C and D). Please criticize those options and your rationale over them. Why you like some and why you dislike others.

They are:

A. Provide @module_visible_to annotations with best-effort warnings

B. Provide defmodulep where modules must be explicitly required+aliased and it will error if you break its boundaries. The namespace the module will be assigned to is private, which means you have no official ways of accessing a private module beyond its original intent.

C. Provide defmodulep where modules must be explicitly required+aliased and it will error if you break its boundaries. However, the namespace the module will be assigned to is public, which means you can access it directly, without any visibility check, by using its long name. For example, defmodulep Foo.Bar would be accessible directly via :"Elixirp.Foo.Bar".

D. Provide defmodulep where modules must be explicitly required+aliased but it warns instead of erroring if you break its boundaries.

If you can think of other implementations and approaches, please drop a comment to so we can amend the proposal accordingly.

Thank you!