Proposal: Private modules (implementation specific) (closed)

I have mixed feelings about it. On the one hand private modules are nice idea, on the other hand I think that this proposal is quite “too much”. What I would like to see instead is rather compiler warning and module attribute (maybe even with EEP), so this would not be a hard error, but compilation warning, something like:

defmodule MyApp,Private do
  @private [MyApp]

  def hello do
    IO.puts "hello world"
  end
end

This would be less strict than Your proposal, but I think it would be more “in line” with rest of the platform.

This is really interesting idea!

I’m not sure about it. On one hand it could be probably nicer for iex sessions (not sure about it), but on another I would like to see a module which is only accessible for specific library/scope.

What would happen if somebody would do:

defmodulep NotMyLibrary.Private, visible_to: [NotMyLibrary] do
  # …
end

and then I would do something like:

defmodule MyApp do
  # …
end
# …
defmodule NotMyLibrary.SimpleHack do
  require NotMyLibrary.Private, as: Private
  # …
end

Is it possible to add something like:

defmodulep NotMyLibrary.Private, ensure_app: [:not_my_library], visible_to: [NotMyLibrary] do
  # …
end

so in case of compile time definition of NotMyLibrary.SimpleHack in my_app app we would receive error/warning like there is one on overriding modules.

Even if DDD is different for each compilation then it’s possible to find it like:

defmodule Example do
  def sample(module) do
    source = Keyword.keys(:code.all_loaded())
    regex = ~r/\Amodulep_[0-9]+_#{to_string(module)}\Z/
    matcher = &String.match?(&1, regex)
    source |> Enum.map(&to_string/1) |> Enum.find(matcher) |> String.to_existing_atom()
  end
end

real_module_to_alias = Example.sample(Foo.Bar)

Of course it’s not as safe as depending on official way, but I believe that in such way we could guess real full module atom. Hope I did not missed anything here.

I have also two questions here:

1. Would be there easy way to use private modules inside iex session? It would be awesome if it could be possible only for iex session started only for restricted app.

2. How Elixir (and therefore ex_doc) would handle @moduledoc and @doc attributes inside private modules? I see here a nice win-win to close @docp and related topics.

It would be awesome if it will be introduced into Elixir core as people would be forced to stop depend on private APIs.

This part might get affected, like if you iterate using the repl to validate quickly a function that now is on a private module. Unless given an option, mix could compiled your modules to be visible to iex if you are in dev or test env.

I think most of the languages that has a VM or interpreter can be monkey patched somehow. Like in Java you can call a private method or read a private property using reflection, and also I’ve seen people creating the same package structure alongside your’s, to emulate the dependency one so you could extend a protected class.

While I understand the technical reasons for doing so, I dislike the required alias in require. We generally do not alias things for greppability reasons, so having a way to be able to use the FQN would be nice. Up until now, alias has been opt in and composes independently with other features but this requires agreeing with two concepts to use one.

I find myself not wanting this change. It feels like a lot of special cases, for mild protection. I feel like people who use code with something like @moduledoc false on it today know what they are doing and will not be deterred by this feature. The protection is easily defeated as shown in this thread.

Maybe I just haven’t felt the pain this feature is targeted at though. Perhaps a good example use case would help me see the value.

I agree with everything @JEG2 said.

I don’t know anyone who wants to call a private API; I know a lot of people who need to call one. I’d prefer a warning sign rather than a guard dog. For me, @moduledoc false/@doc false have been enough.

I’m generally for this feature since I think the latter case leads folks to reaching into private APIs out of “necessity”, where they would otherwise ideally engage with library authors and team members on required API changes to accomplishes their needs. Over time these kind of quick fixes bite folks and maintainers alike, so if things could be locked down as private I think it would stop a lot of pain in the long term, for maybe some effort in the short term.

I’ve seen it both ways. Some folks simply are simply unaware or miss the doc notation and blissfully call APIs they shouldn’t, while others go rouge and intentionaly break the rules

I think this feature would help in both cases. Newcomers wouldn’t accidentally ship brittle code against private APIs and more seasons devs would avoid the lazy trap of calling the API because they can.

A common implementation is to have the visibility of your modules associated to the “idea of a package” but Elixir does not quite have the concept of a package. Elixir does provide the idea of “applications” but they are define only after the code is compiled. That’s why the “package” approach has been ruled out in favor of a explicit visible_to control.

I think a mix project is roughly analogous to a package in this case: a bundle of related code files, so restricting visibility to just the project (and by dropping the visible_to attribute) would be OK.

That would remind me of the internal keyword used in C# (class visible in current project/assembly only). Since it’s common in C# to create a separate assembly for tests, you would often declare internals visible to: other_assembly to be able to test the functionality.

As for the main question, adding private modules I dont think will make things any clearer, it could just lead to a bigger language surface without clear benefits.

Again, from my experience coding in C#, we’d often come across library code that we had to extend but the authors decided that their code should never be extended (private/final classes etc.). Since making things less accesible makes more sense in library code (which is meant to be shared) we often had to find workarounds (reflection, decompiling, forking repos etc) just to be able to do our work.

All in all, I think the potential for confusion and abuse is there and it’s real, and I can’t say I’ve missed any of these things in Elixir. Perhaps a warning like others said is more than enough.

Answering many comments at once…

First of all, in regards to the feature as a whole being necessary: it absolutely is. For an example, just look at the Elixir v1.7 release which broke many packages that were using Elixir private modules. When talking to developers who were using these APIs, most of times, they simply did not notice it wasn’t documented. On large systems, this leads to a cascade effect that makes it very hard to update only parts of the system, because in order to update Elixir, you also need to update dependency X, which may break Y and Z due to private APIs, and so on.

In regards to the feature having workarounds: yes, there are workarounds and even simpler than the ones posted on this thread. But let’s be honest here: there is no implementation that will forbid someone from bypassing the visibility boundaries if someone really wants to do it. All languages that I have explored while writing this proposal has this “flaw”. In a nutshell, we mostly need a better way to document intent.

That said, I think @hauleth does provide a good point: all of the features above could be achieved with just warning. So we need to do a choice between a hard failure (this proposal) or a warning (as mentioned by @hauleth). Implementing it via a warning would be much simpler but such warnings would be a “best effort” and they would be quite easy to bypass. For example, by doing:

mod = SomePrivateModule
mod.foo()

Is the warning worth it even if it is not guaranteed? Thoughts?

The compiler has no idea about applications. Apps are purely a build tool concern that are assembled after all modules are compiled. In a way, this is great, as the language core is small and it builds new concepts on top of existing ones, but it means this feature (and modules in general) cannot integrate with applications.