I think this is a great proposal! It’s a chance to be more explicit about which parts of codebase are the public entry points and which parts are just the private implementation details.
It’s a much more effective and intention revealing construct than @moduledoc false.
Currently, within teams/companies, developers may agree on certain conventions, such as "let’s use the public interface of module A, instead of calling A.B.C.some_function directly from D". But it requires discipline and before you know it your code is tightly coupled and it becomes hard to reorganize or refactor those internal (private) modules.
This proposal also goes hand in hand with Phoenix Contexts: we “know” that we’re not supposed to call directly call MyApp.Blog.Post.changeset from the controller. But it is still possible and currently there is nothing in the language that prevents it. So it would be great to be able to be more explicit about it via a defmodulep, similarly to defp.
Regarding option C:
Would specifying visible_to be required? I think in almost all cases the value of visible_to: is chosen to be that of the namespace one level above. It almost seems redundant. In fact I think it’d seem odd if visible_to: would point to entirely different namespace (i.e. MyOtherApp). If I understand correctly it is not desirable/feasible to dynamically deduce visible_to automatically when not supplied.
I dislike implicit naming conventions in modules (like what Phoenix does to match controllers and views). I’d prefer if this was as explicit as possible.
I would argue that this isn’t a technological solution at all, but exactly what you advocate. It’s a communication tool designed to help with “people problems”.
And as far as learning, most people are accustomed to working in a language that has some form of private/public parts in it. If they then learn how Elixir does it currently, it would seem like nothing short of a really hacky solution.
Again, this proposal isn’t here to stop determined attempts to use private code as it were public. It’s about the clear expression of intent. I just don’t understand how some people see this as a negative. Conventions and warning are at worst hacky, at best easy to miss or ignore.
This isn’t the only problem this proposal solves. As many others have said, even within the same application, maintained by the same team, this proposal helps outline boundaries for future uses of the code. Edit: I have updated the proposal to list this as well.
Also, I have had in multiple instances to maintain code that I was really not supposed to, nor didn’t want to, to not break other people’s code that were accidentally using private APIs. It is easy to see how this is not healthy in the long term: it discourages the team to bring more improvements while increasing the maintenance burden.
So saying this only aids with the public perception is not true. Especially because, it is not about even about perception, since those applications are actually broken on updates due to transitive dependencies. This is mentioned directly in the proposal: “Even more worrying, is that this practice in the long term can be really harmful as systems grow in size”.
Isn’t improving the language so we can better communicate between teams and users about our boundaries a way to use technology to improve what is inherently a people’s problem? What are the other suggestions then to solve this problem? How can we better reveal intent about the use of “private” modules given the current mechanisms are clearly not enough?
I’m not sure I accept the original premise/reason for this proposal. I understand the desire to have a demarcation between the exposed API and internal implementation. but I prefer to rely on the structure of the library and it’s documentation rather than forcing something explicit. I like the fact that everything is public. Most libraries are structured with a main module with the exposed API, all other modules are implementation. If I use something in an implementation module I do so with the understanding that my code will break in the next version and I’ll need to fix it.
As to the issue if Elixir is updated and breaks package X, which then breaks package Y and Z, I’m okay with that. Package X will get updated and package Y and Z will get updated. If one of them doesn’t, say Y, then folks will either stay on the previous version of Elixir if they HAVE to use Y, or package A will come along and replace Y. This is just the natural state of Open Source Software.
Having said all that, I’d prefer option D then C if I have to choose one.
There are literally maintainers of the biggest and most used Elixir packages in this thread who tell you that this is not working. I sympathize with the sentiment but practice shows that many people simply rush their way through tasks and do not notice these nuances.
Agreed. But that’s not helping corporate teams. We have enough work as it is and we can’t be eternally tracking The-New-Hotness®. We have stuff to do. If we cannot keep on top of things by re-reviewing the ecosystem once a month then we’ll eventually move our business elsewhere.
I’ve been thinking and maybe my main objection is that any package that used defmodulep will be instantly impossible to use in older Elixir versions. Because defmodulep is a macro, you can’t simple define it in a dummy module somewhere in your app, and just use it. You have to import your dummy module, and that propagates through your source in a distracting way. But that’s the cost of innovation, I guess…
The more I think about this proposal, the more I think there should be both a requirep and an aliasp macro. An aliasp macro would play well with the idea that private modules should start with :"Elixirp." and would avoid the cost of requireing the module (because alias calls can be made in parallel).
Even if we decide to have a stronger level of name mangling, such as starting the private module names with :"Elixirp.<hash_of_the_module_name>.", aliasp would work as long as the hash is deterministic. So no need to require the module anyway.
Another possibility would be to rename aliasp as something like expose. That way it would read quite naturally:
defmodulep MyLib.Hidden do
@visible_to: [MyLib.Public]
# ...
end
defmodule MyLib.Public do
expose MyLib.Hidden
# or
expose MyLib.Hidden, as: Hidden
# instead of
aliasp MyLib.Hidden
# ...
end
require could continue to be require instead of requirep in this case.
But ultimately, I guess I prefer the consistency of having everything end with a p: defmodulep, aliasp, requirep, defp, defmacrop, defguardp, no matter how bad it sounds while reading these aloud.
So I’d vote for defmodulep, requirep and aliasp.
I still prefer the mangled module names instead of the ones starting with the fixed Elixirp prefix, but for practical reasons I might settle for Elixirp.
As I said, I fully accept that we will end up with private modules in Elixir. I have no illusions that I will change anyone’s mind. I continue the conversation only for those who wish to understand the contrary opinion.
Within a team, if some part of the code they own is dependent on some other private part that they also own, then it’s up to them to resolve that before refactoring the private part. If/when a team chooses to try to prevent this situation, then they have choices like pair programming, code reviews, linting tools, etc.
Effects within a team are less important than third party dependencies though. I’ve lived through projects with bloated node_modules, 1000 line Gemfiles, and heavyweight maven dependencies. The more of those you have, the harder it is to upgrade. And, yes, I’ve even debugged into some of those libraries that intentionally work around the language’s privacy features. The more dependencies you have, the greater the chance at least one of them has something like this happening. Private modules will change how frequently it happens, but it does not completely prevent it.
Third party libraries are not “free as in beer” but more like “a free puppy.” Puppies are great! But, you may not be able to make impromptu trips out of town any more because the dog needs to be fed, or you may have to buy new shoes because the puppy chewed them, etc. The point is that third party libraries change your “lifestyle.” Teams should take a more cautious approach to using them than is the norm today. This is the direction toward reduced brittleness in software in my opinion, but I’m still trying to figure out best how to encourage it. Thinking that private modules solves the problem once and for all works against this goal. I’m not suggesting anyone specifically thinks that, just that it would be contrary to my goal.
I didn’t mean to imply perception was the only value, just that it was the primary one in my mind. Elixir is a growing community, and developers have plenty of other choices of language communities to invest their time in instead. We’d like Elixir to reach a critical mass that we can continue (or begin to) enjoy our time working in this wonderful language professionally. If people who are considering investing time with Elixir hear about too much brittleness it might send them elsewhere and limit that critical mass.
I wish I had a great answer, but I don’t. But my main point is this is not so clear to me. And on top of that, I’m not sure we’re weighing the costs of the feature enough. Aside from what I mentioned earlier about needing to learn and use it, what about operational debugging when calling a private module is a huge help, or how sad will it be to announce the compiler has slowed down x%, or what if OTP introduces a low level feature in the future that would give us an even better way to communicate privacy?
It boils down to how we weigh the “current mechanisms are clearly not enough” with the potential intangible cost of private modules in the long run.
So, I’ve shared my opinion. I don’t have the time or energy for a big debate. I"m not upset about it. I expect to see private modules in Elixir, and I expect I’ll grumble a bit to myself but mostly my life will go on with few changes.
I don’t think this is a big concern because that’s what happens on every new Elixir feature. with, DynamicSupervisor, new child specs, the new struct inspect in v1.8, etc. The author of a library needs to decide which versions they are support and decide to use an Elixir feature accordingly.
Yup, yup! If we go with a separate command, aliasp is enough and we don’t even need requirep. The verification can be done in parallel too if we defer it to until the module is compiled (i.e. in an after_compile callback). I am still worried about adding yet another “instruction” though. People already confuse use, require, import and alias.
Thanks @gregvaughn! One last question, if you have the time. Given everything you said, how does proposal A fair? Since it doesn’t really introduce an operational cost, the implementation is simpler, and it can improve the communication between teams?
You are way ahead of what I was thinking xD I was thinking of raising the error when a function from that module was used. Raising the error at compile time is pretty cool, and if it’s possible I’m in.
I’m the first to admit that this is very confusing. But adding aliasp, requirep or even importp and usep wouldn’t increase complexity, because anyonw who uses the first four immediately understands how to use private versions. They’re just the same but for a private module.
The original confusion comes from the large number of concepts you need to understand, some of which require the user to be compfortable with macros and (even worse!) the implementation details of the Elixir compiler. For example:
to understand use, you need to know what macros are and how to define them. Otherwise, use is complete magic. It doesn’t help that most educational materials tell you to use GenServer, and Bam! you have a genserver. When you understand how macros work, you understand that the use macro is defining a bunch of overrideable functions in your module and implementing the GenServer behaviour. But notice that there are a bunch of concepts you need to understand (macors, overrideable defs, behaviours, etc.)
to understand the differences between require and alias, you need to know that the Elixir compiler is a parallel compiler. And that is because it is so slow that it’s worth it to hack the default Erlang error handling mechanisms just to improve compilation speed. So you have a distinction which seems arbitrary until you understand the reasons behind it. If the elixir compiler was blazing fast (which unfortunately, it isn’r for a number of reasons), you could require everything and alias nothing. Ultimately, alias is just a faster require which can’t export macros because they haven’t been compiled yet.
To understand import, you must again understand that the Elixir has a prallel compiler and you can’t assume that the compiler for one file knows what exactly has been defined in another file, unless you force an ordering.
I’m totally unaware of Elixir’s early history, and these decisions probably predate the parallel compiler, but these are the ways I personally make sense of the use, require, import and alias. If I were designing the language from scratch, I’d have only import X and import X, qualified: true or something like that. And use, of couse (actually, I’d have never come up with use, which is an amazing feature to save typing). But ultimately, use X is just import X, qualified: true; X.some_magic().
This complexity makes people confused in the beginning, of course, but it’s pretty simple when you think of it as a trick to improve compilation speed.
EDIT: This goes offtopic by a wide margin; the main point is that what whoever understands user, … require, will also understand the p versions without problem, because they are just private versions of the four commands.
Perhaps before private modules are introduced in Elixir, the core Elixir maintainers should touch base with the Erlang/OTP team first? Maybe they have something in mind as well.
You need requirep if you want to use macros from a private module of course. And if you’re going to add both aliasp and requirep, you might as well add usep and importp for completeness.
They do. I don’t want to side-track the discussion but they do predate it as they do four distinct things. We had countless discussions to unify this and the consensus was always that, even if we can hide behind a single API (like Python), they still do distinct things, and the different things are the source of confusion, not the API (as we see similar issues in Python).
You could alias first and then do the rest using regular APIs. In this sense, maybe having a separate thing like expose would be better. Unsure. Your point is also valid.
Thanks!
To be very honest, this is extremely unlikely. This change is far com a consensus here and the Erlang community is even more conservative than us. There is also an old-ish proposal for a similar feature for Erlang as EEP 5.
People already confuse use, require, import and alias.
