Proposal: Private modules (general discussion)

official-proposal
#48

Since private modules are deemed a necessary addition, I think option D is the best compromise

  • Warnings instead of hard errors
  • The require+alias macro makes the feature perhaps more discoverable and more explicit than A
0 Likes

#49

I guess I don’t see how any of these solve that problem. My guess is most people indirectly use private APIs indirectly via a library. So, when they upgrade Elixir and that library breaks, the assumption is Elixir broke it. If people are directly using private APIs and not realizing that they’re doing so, I could see what you’re saying being true.

0 Likes

#50

It is a combination of everything said.

Many developers don’t think about boundaries because the language has no constructs to say so (besides @moduledoc false and @doc false, which are minimal). This means people don’t write annotations, which means that even if a developer would respect those boundaries, the boundaries are not there. And similarly, because those boundaries are not easy to check, you may accidentally cross them, even when they are defined.

For example, imagine you want to provide a reduce_with_super_power. You may do so by copying Enum.reduce source and then changing it a bit. The problem is that Enum.reduce called a private module, but you didn’t bother to check everything the code was calling when you copied that small snippet, and now your code may break in the next Elixir release.

This is just one example. Working on Ecto, Plug and Phoenix I saw it happening in other situations too. However, if someone knows that something is private and they decide to call it anyway, then there is nothing we can do. But at least they have been told so.

In general, it is about intent and communication and it is really hard to show intent and communicate when you have limited constructs to do so.

8 Likes

#51

I have some complicated formulas (ex: related to daily items average pricing calculation, weekly automatic discount placement, etc) in some modules that are only used by 1 or 2 modules. One These complicated formulas need to be splitted in different modules (for readability and testability), probably 3 to 5, one of them has 8 modules. Do I need to make them private?

Would hiding them be beneficial to my code? I haven’t tried it. But for sure, if I hide those modules, I only have to provide the public modules to be visible for other components to use. So this is much like the context principle in Phoenix? Access some functionalities only from its boundary? :thinking: I kinda think that this private modules (if implemented) will push us to write smaller modules in the future.

Would this become confusion to my other components as well? Because I already have some public and private functions in some modules. But surely, this will change how I test my Elixir code if want to utilize private modules.

0 Likes

#52

Does anybody have an example of private modules in other languages? Especially functional ones.

Maybe we could learn from them.

0 Likes

#53

Yup! I am interested in exploring the impact this feature may have in the language in terms of code organization and documentation. As @imetallica said, defmodulep really isn’t a good name, so maybe we can find a name that is more intention revealing and will guide us in the matters of code organization.

0 Likes

Alternative names to defmodulep (split thread)
#54

One of my friend said he has 18 modules that have only one top function to call in order to call 18 of them all. It was a huge giant module that was later refactored into 18 (soon to be 20) small modules. He said that each of those 18 modules have its own test file and live inside our CI/CD pipeline.

We still don’t know if those 18 modules should be put under private (if module private is going to be implemented), and we still have no idea on how to test them once they gone private.

0 Likes

#55

If you are willing to experiment, you can give it a try now using the POC from the previous discussion: https://github.com/josevalim/defmodulep

The README also talks about testing but it should be mostly transparent. You can also bypass visibility for testing by using the qualified name. It is all in the README. Please do report your findings!

3 Likes

#56

I can not tell about much, but the last time I checked the issue in haskell (which has proper namespaced modules), it was common to have an INTERNAL-namespace directly beneath the main namespace of the library.

There was only this convention, but much more obvious and discoverable than to have @moduledoc false. Especially when reading foreign code its more obvious that a certain piece is meant to be private.

I have no clue how it is done in other functional languages.

Another system I’m aware of, not in a functional language though, is gos. There you name a package internal (IIRC) and all its childs will only be accessible by childs of its parent. Due to the way how gos packages are namespaced (the source repository is part of the name) you can’t even mimick to be in that package. (Well, one could with abusing dependency vendoring, and then you’ll know when you update anyway).

In Java and C++ namespaces/packages can not be hidden, only their content. And its easy to open up even foreign namespaces and “impersonate” to get access.

So to be honest, I have not yet seen a system that realy makes privte stuff private, but I’d prefer an explicit “I am privat, you shall not use me” over a “I’m having no documentation, so please pretend you haven’t seen me”.

3 Likes

#57

One thing that is conspicuously lacking from this entire thread is any discussion of how private modules might be tested.

As @dimitarvp alluded to, there is often a reflex to say “don’t test implementation details”, but when decomposing complicated functionality among several smaller private functions I’ve often regretted not being able to test them. I can easily imagine that I’d want to test the code in a private module as much if not more.

How would testing relate to the four proposals?

Edit: Or at least conspicuously lacking until the last few minutes while I was writing this! Still, I would like to know how testing would work in each of the four proposals.

3 Likes

#58

You still can test them when they are private, just make sure that your tests have the correct alias-prefix.

Also similar to the proposed iex helper, some ex_unit helper would be thinkable as well, but I’m not sure if that is really necessary.

Last but not least, is this a library or a standalone application? If its the latter, hiding the modules might not be necessary at all. If it is a library though, I’d hide the modules by either @moduledoc false or the proposed defmodulep.

0 Likes

#59

I think this can very quickly turn into a game of whack-a-mole. First iex, then ex_unit, then who knows what else In-The-Future®.

Wouldn’t having a function like Module.get_private/1 that knows exactly how to find and return the private module be a better option?

1 Like

#60

I do not think so. But perhaps a :"Elixirp.Module.Private".get_private/1 which is declared in a way that iex and ex_unit only need to prepare wrappers? Then it won’t get out of sync if mangling or other details change.

If we expose that function directly, nothing (or not much at least) would change from where we are now, as it is still to easy to circumvent.

But of course, in ex_unit it should be never necessary, as you can easily create the test-module in a namespace that has access to the hidden/private module.

1 Like

#61

I think it would be a very welcome feature, currently it’s about discipline in any bigger app, so I’m excited about this proposal.

I like option B the most but would also be pretty happy with C.

0 Likes

#62

Will do.

0 Likes

#63

I don’t disagree. But it still requires you to do something special in order to circumvent the private nature of the module anyhow; only it will be a well-known mechanism in the stdlib API and it will have better readability.

50/50 though, would not insist on it. Plus adding stuff to the stdlib is a really heavy-handed measure, that much is true.

1 Like

#64

About point A:

I’m not sure if this is a good idea at all but, what if instead of being an exposed module, we inline all the functions/macros/etc… of the private module, inside the modules listed in @visible_to: [Foo, Bar, Baz]? Is that possible? Also, how could we guarantee that the use X from the private module doesn’t conflict with the use Y that uses the private module?

0 Likes

#65

It’s just some libraries in our app, not a standalone app. Each module has its own test file. Just functions grouped by modules, nothing really fancy. But because they contain some arcane formulas, readability and testing is really necessary here for each module. We just couldn’t test only the top calling function and expect some magic to work.

Anyway, I also work in Ruby on Rails code in my fulltime job, having service pattern (something like this: https://hackernoon.com/service-objects-in-ruby-on-rails-and-you-79ca8a1c946e) and form classes really helps me and my team to do testing and splitting the code for maintainability, it has been a breeze. But these service and forms classes are public, and they are reusable for anyone. (But this pattern also raises some significant problems that I will explain below.)

What comes to my mind, is that Elixir private module is a sort of contract for code reuse in internal app/library, but not for global/public. “You want to use this? Well just whitelist/make visible the caller.” This is probably intended for several people working together in a close ‘vicinity’. Probably the one who create a private module will be the one who will call (whitelisting) it as well. Something like: Jose created a private module, the farthest people who is going to call it is probably Chris McCord, not me, not you, not most Elixir users. CMIIW.

I’m kinda thinking that Elixir private module (if it is implemented) is best used for hiding inner complexity for libraries. “Yes you can test it, but please don’t call it if you can. Let us handle and maintain this private module. You don’t need to. You don’t have to.” This is in contrast with above service pattern, where everyone can use it, and hence when somebody needs to change (or hack it :dizzy_face:) it, it will affect all other codes which call it, it will end up with many flag variables if we don’t immediately refactor/duplicate it.

Now after some moment of thinkings, this private module proposal made me (and my friend) to rethink “among 18 modules that we got here, which one should go public for other developers to reuse? Should they be reusable? :thinking:”. This in turn raise a question to me, which module should go public and reusable, which module should be private.

0 Likes

#66

I like option C the best.

1 Like

#67

Option C for sure, for the ergonomic and pragmatic reasons described earlier. I like the addition of the feature generally as well.

1 Like