PSA: Do not use private APIs, request a feature instead

Elixir News & Announcements Elixir Notices
1

Hi everyone,

With the v1.7 release, we have received reports of some dependencies no longer working properly on the new release.

Developers should not expect breaking changes from Elixir on minor and patch releases. We outline our compatibility and deprecations policy in our docs: Compatibility and Deprecations — Elixir v1.15.2

This means that, if there is something effectively broken in a new release, it is one of:

  1. We messed up and accidentally introduced a regression, which we have to fix
  2. We fixed a bug that you were accidentally relying on. It is still our bad. We will either have to accept this or find a better way to do a bug fix (for example, introduce a new function and deprecate the old one)
  3. Some project (it could be yours) was using private API

Thanks to everyone, we are able to catch the huge majority of the occurrences of 1 and 2 during the RCs. However, there are still many cases where developers are simply using private Elixir modules and functionality - which may be renamed or completely removed in new Elixir versions. We do not provide any guarantees for private functionality. Needless to say, DO NOT USE PRIVATE APIs.

Using private APIs generate a negative cascade effect in the community because it makes developers unable to update to the latest Elixir version.

How to help

If you are a user of a project and you notice it depends on private functionality, consider reporting a bug or submitting a pull request.

If you are a library author and you are using private APIs, don’t. :slight_smile: If Elixir has a functionality that you need but it is private, please open up a feature request to make the desired functionality public.

Also, consider running your CI builds against Elixir master. It is as easy as:

- wget https://repo.hex.pm/builds/elixir/master.zip
- unzip -d elixir master.zip
- export PATH=$(pwd)/elixir/bin:${PATH}

Here is more info on available precompiled Elixir versions, branches and tags: GitHub - hexpm/bob: The Builder

While I wrote this thinking about Elixir, it likely applies to most libraries out there.

29 Likes
3

It might be nice to give people a reminder about the @moduledoc false convention. Or even better a tool that flags uses of private apis from the standard lib as a compile time warning.

6 Likes
4

Good point @bbense! We talk about @moduledoc false in our documentation too: https://hexdocs.pm/elixir/master/writing-documentation.html#hiding-internal-modules-and-functions

1 Like
5

Also, it would really be nice to have at least a warning when a private API is being used by a module outside its application. I remember that being discussed somewhere else but don’t remember the outcome of it, maybe that’s not even possible, IDK.

6 Likes
6

Wouldn’t it be better to be able to have truly private APIs built into the language so that compilation against “private” modules and functions would fail? Seems like something the language owners could add.

Something like a defpmodule macro that accepted either as part of the module def or possibly a well known constant, a list of module friends able to access internally public functions.

4 Likes
7

Yes, that would be the best approach, although it is not trivial to implement because of how modules are dynamic. Unless we take some liberties, such as, private modules always need to be required before using or something of sorts. Somebody would have to write such proposal. We should do it in another thread though.

2 Likes
8

Maybe, if you want to catch things like String.Tokenizer.tokenize(list). But if you’re happy with only detecting modules that are imported/used/required/aliased, then it’s pretty easy:

defmodule AvoidPrivateAPIs do
  require Logger

  def hidden?(module) do
    match?({:docs_v1, _, :elixir, _, :hidden, _, _}, Code.fetch_docs(module))
  end

  def private_module_message(verb, this_app, that_app, this_module, that_module) do
    """
    The module #{inspect(this_module)}, which belongs to application #{inspect(this_app)}, \
    has tried to #{verb} the hidden module #{inspect(that_module)}, \
    which belongs to application #{inspect(that_app)}.
    """
  end

  defmacro import(quoted_module, opts \\ []) do
    {that_module, _} = Code.eval_quoted(quoted_module)
    this_module = __CALLER__.module
    this_app = Application.get_application(this_module)
    that_app = Application.get_application(that_module)

    if this_app != that_app and hidden?(that_module) do
      message = private_module_message("import", this_app, that_app, this_module, that_module)
      Logger.warn(fn -> message end)
    end

    quote do
      import(unquote(that_module), unquote(opts))
    end
  end
end

defmodule PrivateAPI do
  @moduledoc false

  def hey() do
    "Hey!"
  end
end

defmodule TryToUsePrivateAPIsFromOutsideTheApp do
  require AvoidPrivateAPIs
  # Will log a warning, because String.Tokenizer is defined in the :elixir application
  AvoidPrivateAPIs.import String.Tokenizer
end


defmodule TryToUsePrivateAPIsInsideTheApp do
  require AvoidPrivateAPIs
  # Will not log a warning
  AvoidPrivateAPIs.import PrivateAPI
end

The example above shows only how to deal with imported modules, but the idea might work for other kinds of modules. There are probably some subtleties I’m not capturing here, of course…

9

I would say some of the issue stems from the fact that its not easy to be able to identify if a module is intended for public consumption or not. The elixir language is well documented and that makes it clearer, but it doesn’t make it black and white. How should one tell the difference between “maybe its missing documentation” and a module which is intended to be private?

Theres a larger issue to solve here, as has been mentioned. However, to address the problem at hand in the moment perhaps all internal modules should be marked as such. String.Internal.Tokenizer would make it plainly clear.

1 Like
10

Shouldn’t @moduledoc false just become the default behavior for undocumented modules? I’ve always wondered why documentation has been considered opt-out in ex_doc

1 Like
11

For the Elixir language itself it should be black and white. Elixir categorically marks all private functionality with @moduledoc false and @doc false. Those do not appear on the official documentation, which means their usage is being lifted directly from the source code.

To clarify, this has never been an issue in Elixir itself. Or we explicitly document it or we tag it as false. Sure, there are projects in the community where this issue will arise but it is definitely not the source of confusion in this case.

3 Likes
12

@josevalim: Please take a look at @lexmag response:

It was written in almost 2 years ago and still ExUnit.Diff has no public API. It would be helpful if you will be able to change it.

1 Like
13

Thanks @Eiji! This is a good example. I recommend you to start a discussion on Elixir issues tracker. We do not track comments or discussions on the forum as feature requests.

Note that it is unlikely that exporting ExUnit.Diff as is would be useful to you, as the current algorithm is recursive on itself, and you want to support more data types than the ones we do. There are different ways we could solve this, for example, by making the recursive function an argument on ExUnit.Diff, introducing a protocol, or by breaking ExUnit.Diff into smaller, reusable bits. So when opening the issue, please make sure that you provide a more complete proposal with examples on how you believe the ExUnit.Diff should be exported.

3 Likes
14

@josevalim would it be acceptable to detect the use of private modules only at the level of use/alias/import/require like I’ve done above?

I dont think you can handle the other cases (the use of modules that aren’t even aliased) without runtime checks.

15

I think this is a good example of why the pragmatic version of “Don’t use private APIs” is “Be aware of the risks of using private APIs”. Any feature that provides some kind of compile time check on private API usage should have an escape hatch, because there are situations where being locked to a specific version of the language and having a feature is sensibly considered a worthwhile trade off.

A tracking mechanism for private usage would actually make that sort of thing safer, since it’d be easier to audit.

4 Likes
16

While you really can’t stop someone from using “private” modules in the BEAM, you could do better compile time warnings by adding a new module attribute @use_in which has a list of modules in which the the “private” module is intended to be used.

This has the benefit of being explict.

17

That’s exactly what my code snipped above does, except instead of declaring explicitly where the module should be used, it assumes the module should only be used inside its application.

The big decision here is not how you specify where you can use a module, is whether catching only the places where it’s used, imported, etc is enough.

18

PS: I really like the @use_in idea. The only problem with the attribute name is that in elixir, use is a very loaded word.

1 Like
19

Wouldn’t it be more clear to explicitly tag it as private or unstable? Possibly with additional scary colors and “user beware” attribution in generated docs. This communicates exactly what you mean. It lets projects document private or unstable classes as well.

There’s a way other than @moduledoc false and @doc false to prevent docs from showing up in public distributions isn’t there? Like an exclusions list config for the mix task or something.

20

I know this is going to be a long shot, but has anyone considered releasing new versions of Elixir in a way similar to how the Rust team does it? They download and compile every single package on crates.io before they do a release. Not only does this help them make sure they don’t break things for everyone each release, but it also brings to light bad practices the community may be engaged in, like using Private apis. Findings like that can be discussed early on either to inform the community that they aren’t squared away, or to inform the dev team just how far reaching their changes really are.

If y’all are already doing this I apologize for being so far behind.

3 Likes