Under Elixir 1.15, `unicode` lib dependency only works after first compile

Questions / Help
, , ,
1

This one has me scratching my head.

I author the table_rex library; which is a little tool which outputs nicely spaced ascii data tables at the command line.

I recently accepted a PR to enable support for languages other than latin (East Asian ones in particular which come with varying character widths); and this PR came with a new test only dependency named unicode.

I checked the code out and mix test completed after pulling deps, so it went in; however, this I then noticed that if I ran mix test again…some of tests fail due to an undefined helper function:

** (UndefinedFunctionError) function Unicode.EastAsianWidth.east_asian_width_category/1 is undefined (module Unicode.EastAsianWidth is not available)

If you then do the following:

mix deps.clean --all
mix deps.get
mix test

The test suite will pass again. If you run mix test a subsequent time it will fail with the undefined function error.

I have reproduced this behaviour on this branch, which deliberately runs the test suite twice in succession. As you can, it passes fine on Elixir 1.13 & Elixir 1.14, but fails on Elixir 1.15 (with both OTP 25 & 26).

Does anyone have any ideas what could cause this, or has anyone seen behaviour like it before?

Thanks in advance for any help.

As an extra…

This behaviour is also true of iex. After the initial compile of unicode it’ll work fine - but subsequent shell starts will result in the same function being undefined. The shell history of this:

mix deps.clean --all
mix deps.get
MIX_ENV=test iex -S mix
~/Source/table_rex misc/test-ci* 19s
nix-shell ❯ MIX_ENV=test iex -S mix
Erlang/OTP 25 [erts-13.2.2.1] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

==> unicode
Compiling 20 files (.ex)
Generated unicode app
==> table_rex
Generated table_rex app
Interactive Elixir (1.15.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Unicode.EastAsianWidth.east_asian_width_category(36984)
:w
iex(2)>
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
^C
~/Source/table_rex misc/test-ci* 32s
nix-shell ❯ MIX_ENV=test iex -S mix
Erlang/OTP 25 [erts-13.2.2.1] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Interactive Elixir (1.15.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Unicode.EastAsianWidth.east_asian_width_category(36984)
** (UndefinedFunctionError) function Unicode.EastAsianWidth.east_asian_width_category/1 is undefined (module Unicode.EastAsianWidth is not available)
    (unicode 1.16.1) Unicode.EastAsianWidth.east_asian_width_category(36984)
    iex:1: (file)
iex(1)>
2 Likes
2

Here is a fix: Use extra_applications to compute apps list by josevalim · Pull Request #54 · djm/table_rex · GitHub

How I debugged it:

  1. I ran MIX_ENV=test iex -S mix (since you said it reproduced it)
  2. I ran :code.get_paths and I noticed :unicode was not in the code path
  3. The first root cause is to check if :unicode is listed as a dependency (yes) and it is listed in def application (it was not)

applications: ... requires you to explicitly list all apps. It was the default for new apps in Elixir v1.4 and earlier but since then changed. If you use extra_applications, Elixir builds the list for you. Updating the list fixes it.

Why does it work when you clean it? Because then you compile and the modules are stored in memory, so they are available even if they aren’t in your code path. Since Elixir v1.15 now prunes code paths, they aren’t compiled (so unavailable) nor they would be loaded (load path missing).

7 Likes
3

Thinking about how to prevent / warn around this in code like Compile.App.project_apps and I’m pondering: is there ever a situation where a user would want things in deps that weren’t in applications/extra_applications and weren’t flagged with runtime: false?

It seems like this config should be OK (in a project like the original poster’s):

  def application do
    [applications: [:logger, :unicode]]
  end

or this one:

  def application do
    [
      applications: [:logger],
      extra_applications: [:unicode]
    ]
  end

but not the one with just applications: [:logger]. :thinking:

1 Like
4

@djm, that’s a cool use for unicode (I’m the author). If there’s anything you think it can do to make it easier for you to manage variable width formatting please let me know (here, by DM or even better by opening an issue.

5

If I’m understanding this correctly I’m pretty sure that you would have hit this issue when building a release that included this library. So with the change in Elixir 1.15 you’re hitting that same issue earlier (which I would generally consider a good thing).

2 Likes
6

This is a test only dep, so not in this particular case, but generally speaking you are correct!

2 Likes
7

And in one sleep, I’m reminded of why the Elixir community is so wonderful.

Frustrating though as I knew all this, I’ve just been in Python land recently and forgot about application config quirks. This library harks back to Elixir 1.1 so that fully makes sense.

Very appreciative of the steps to debug it @josevalim; thinking of code_paths like PYTHONPATH would have let me solve this easily.

Thanks all!

2 Likes
8

and thank you for unicode @kip!

3 Likes