Confusing behavior of `optional` deps in `mix.exs`

I’m struggling with optional: true dependencies in one of my projects. This project will wrap an API (complete code here), and users will be able to provide their own functions/modules to handle the HTTP interaction and response body decoding if they wish. However, default implementations of these will ship with the library.

These default implementations depend on :hackney and Jason. Having learned about optional dependencies, they seemed like exactly what I was looking for (docs):

:optional - marks the dependency as optional. In such cases, the current project will always include the optional dependency but any other project that depends on the current project won’t be forced to use the optional dependency. However, if the other project includes the optional dependency on its own, the requirements and options specified here will also be applied.

(The file is here if you want to poke around.)

If I change

  defp deps do
    [
      {:hackney, "~> 1.14", only: [:dev, :test]},
      {:jason, "~> 1.1", only: [:dev, :test]},
      ...
    ]
  end

to

  defp deps do
    [
      {:hackney, "~> 1.14", optional: true},
      {:jason, "~> 1.1", optional: true},
      ...
    ]
  end

I get strange results. First, dialyzer (used via dialyxir) no longer runs successfully and I get these errors:

:0:unknown_function
Function Jason.decode/1 does not exist.
________________________________________________________________________________
:0:unknown_function
Function Jason.encode/2 does not exist.
________________________________________________________________________________
:0:unknown_function
Function :hackney.body/1 does not exist.
________________________________________________________________________________
:0:unknown_function
Function :hackney.request/5 does not exist.
________________________________________________________________________________
done (warnings were emitted)

Then, 2 tests using hackney fail, although others tests also using hackney still pass. The tests failing are these.

What’s going on?

Per my understanding of the docs, using an optional dependency shouldn’t change any of these results since the dependencies would be included in my project. They simply wouldn’t be forced on the users of my package, and would only make the version requirements apply if those dependencies were already included in their own project’s mix file.

2 Likes

I cannot reproduce these errors. When I set the hackney and jason dependencies as optional tests pass cleanly and I get no errors from dialyzer.

What Elixir version are you using? Try deleting the _build and deps directories and do a clean mix deps.get and retry.

2 Likes

Thanks for your help!

It seems my issues are caused by the :hackney and :jason OTP apps not getting started when configured as optional dependencies. It appears the application inference introduced in Mix 1.4 isn’t handling this case correctly, as it can be fixed by ensuring :hackney and :jason are started (by adding them to :extra_applications). Is this a bug in Mix, or is there some special handling I should be doing for optional OTP apps that should get started if available?


Reproduction

I have pushed a clean branch reproducing the issue.

Env info:

$ elixir -v
Erlang/OTP 21 [erts-10.0.5] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Elixir 1.7.2 (compiled with Erlang/OTP 20)

Steps to reproduce (when on commit e4a85376def2ca2bcb1671ea812714f83b02e132):

$ rm -rf _build/
$ rm -rf deps/
$ mix deps.get
# output omitted
$ mix test
# compilation messages omitted
==> scrapy_cloud_ex
Compiling 23 files (.ex)
Generated scrapy_cloud_ex app
Starting HTTParrot on port 8080
Starting HTTParrot on port 8433 (SSL)
Starting HTTParrot on unix socket httparrot.sock
..................................................................................................................................................................

  1) test request/1: POST requests add included body (ScrapyCloudEx.HttpAdapters.DefaultTest)
     test/http_adapters/default_test.exs:113
     ** (ArgumentError) argument error
     code: test_included_body(request)
     stacktrace:
       (stdlib) :ets.lookup_element(:hackney_config, :mod_metrics, 2)
       /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney_metrics.erl:27: :hackney_metrics.get_engine/0
       /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney_connect.erl:71: :hackney_connect.create_connection/5
       /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney_connect.erl:39: :hackney_connect.connect/5
       /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney.erl:328: :hackney.request/5
       (scrapy_cloud_ex) lib/http_adapters/default.ex:52: ScrapyCloudEx.HttpAdapters.Default.make_request/6
       (scrapy_cloud_ex) lib/http_adapters/default.ex:26: ScrapyCloudEx.HttpAdapters.Default.request/1
       test/http_adapters/default_test.exs:66: ScrapyCloudEx.HttpAdapters.DefaultTest.CommonTests.test_included_body/1
       test/http_adapters/default_test.exs:114: (test)

.

  2) test request/1: PUT requests add included body (ScrapyCloudEx.HttpAdapters.DefaultTest)
     test/http_adapters/default_test.exs:121
     ** (ArgumentError) argument error
     code: test_included_body(request)
     stacktrace:
       (stdlib) :ets.lookup_element(:hackney_config, :mod_metrics, 2)
       /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney_metrics.erl:27: :hackney_metrics.get_engine/0
       /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney_connect.erl:71: :hackney_connect.create_connection/5
       /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney_connect.erl:39: :hackney_connect.connect/5
       /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney.erl:328: :hackney.request/5
       (scrapy_cloud_ex) lib/http_adapters/default.ex:52: ScrapyCloudEx.HttpAdapters.Default.make_request/6
       (scrapy_cloud_ex) lib/http_adapters/default.ex:26: ScrapyCloudEx.HttpAdapters.Default.request/1
       test/http_adapters/default_test.exs:66: ScrapyCloudEx.HttpAdapters.DefaultTest.CommonTests.test_included_body/1
       test/http_adapters/default_test.exs:122: (test)

...................

Finished in 0.8 seconds
184 tests, 2 failures

Randomized with seed 511043

Here’s the root issue:

$ iex -S mix
# compilation messages omitted
iex(1)> ScrapyCloudEx.Endpoints.App.Comments.get("foobar", "1/2/3")
# note: `:hackney.request("https://www.example.com")` would also fail
# debug data omitted

** (ArgumentError) argument error
    (stdlib) :ets.lookup_element(:hackney_config, :mod_metrics, 2)
    /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney_metrics.erl:27: :hackney_metrics.get_engine/0
    /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney_connect.erl:71: :hackney_connect.create_connection/5
    /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney_connect.erl:39: :hackney_connect.connect/5
    /home/david/Documents/dev/scrapy_cloud_ex/deps/hackney/src/hackney.erl:328: :hackney.request/5
    (scrapy_cloud_ex) lib/http_adapters/default.ex:52: ScrapyCloudEx.HttpAdapters.Default.make_request/6
    (scrapy_cloud_ex) lib/http_adapters/default.ex:26: ScrapyCloudEx.HttpAdapters.Default.request/1
    (scrapy_cloud_ex) lib/endpoints/helpers.ex:46: ScrapyCloudEx.Endpoints.Helpers.make_request/1

iex(1)> Application.ensure_all_started(:hackney)
{:ok,
 [:unicode_util_compat, :idna, :mimerl, :certifi, :ssl_verify_fun, :metrics,
  :hackney]}

iex(2)> ScrapyCloudEx.Endpoints.App.Comments.get("foobar", "1/2/3")
# debug data omitted

{:error, %{message: "Authentication failed", status: 403}}
# this is the expected response from the API given the dummy data

If I add :hackney to the :extra_applications in mix.exs, I can run the above mix example correctly and the tests also execute properly.

If I then also add :jason to :extra_applications, the dialyzer errors also go away.

1 Like

The issue here is that optional dependencies are not started by default (this may change in Elixir 1.8) so you need to start the, manually when running tests or doing local development.

The original reasoning for this behavior was that you wouldn’t be able to test how your application functioned without the dependencies if mix always started them.

3 Likes

Is this documented somewhere, or would it be helpful for me to open a PR to add this info to the docs?

2 Likes

No, it’s not documented afaict but the behavior is changed in Elixir master so we can only document on the v1.7 branch.

I found this commit making the change on master. I assume when you said “this may change” in 1.8, it was in the spirit of “you can’t be sure until it’s released”, but that this change is likely to ship in 1.8?

Also I didn’t understand whether it would be helpful to open PR clarifying the docs for :optional in v 1.4 - 1.7, or if in your opinion it would just add noise?

In any case, thanks a lot for your help with my issue!

1 Like

I said “may” because we have noticed that it causes issues when compiling mix projects from rebar3. Others may also rely on the original behavior so there is a chance we will roll it back.

1 Like

Ran into this issue. I had to remove the optional flag from Plug to make the dialyzer happy.

Any idiomatic way to fix such a problem?

1 Like

I’ve solved this by selectively starting extra applications as necessary based on Mix.env():

2 Likes

An even better solution to this is to use the plt_add_apps arg to dialyzer in your mix.exs file:

defp dialyzer do
    [plt_add_apps: [:cowboy], ...]
end

See Use `plt_add_apps` arg to dialyzer to bring cowboy into analysis · phoenixframework/websock_adapter@507229d · GitHub for an example!