Configure Livebook for use behind a proxy

larshei · November 7, 2024, 8:53am

Hey everyone!

I am trying to introduce livebook in a corporate networking setup, meaning all access is going through a proxy that also does SSL inspection.

Preferrably, I’d like to use the standalone Livebook installation for simplicity, but kept running into the following error: Failed to make a request, reason: {:failed_connect, [{:to_address, {~c"hex.pm", 443}}, {:inet, [:inet], :closed}]}

Not finding a way to configure livebook, I built livebook from source, hoping there would be a difference (because the dev setup has proxy/CA settings configured) - but the same error occurs.

I seem to not be able to make any requests from the livebook, as running

:ssl.start
:inets.start
:httpc.request "https://google.com"

leads to the same error.

How could I configure livebook for this network setup (set up proxy and trusted CA)?

jonatanklosko · November 7, 2024, 9:06am

Try adding this to ~/.livebookdesktop.sh:

export LIVEBOOK_CACERTFILE="/path/to/cacertfile"

larshei · November 7, 2024, 9:34am

Thanks for the quick reply!

Try adding this to ~/.livebookdesktop.sh:

Where would that be located on windows?

I tried setting the variable in my console session using $env:LIVEBOOK_CACERTFILE="C:\some\path" and then running livebook from source and also tried to add LIVEBOOK_CACERTFILE in the advanced system settings > Enrivonment Variables > User variables.

The configured value did not get returned when running System.get_env("LIVEBOOK_CACERTFILE") from inside the livebook, even tho just running env in the console prints the variables as expected. ` System.get_env(“HTTP_PROXY”) returns a value as expected.

Stand-alone Livebook was closed and restarted, so was the source-built livebook (and its surrounding shell session).

Edit: Tried a system reboot, no difference. env | grep 'LIVEBOOK_CACERTFILE' shows the value as expected.

Edit: windows-like env access works as well (not sure if linux commands are forwarded to WSL $env:LIVEBOOK_CACERTFILE returns the expected value.

Edit: I created the file, but no change, even tho the file can be read from livebook:

jonatanklosko · November 7, 2024, 10:53am

Where would that be located on windows?

Create %USERPROFILE%\.livebookdesktop.bat with something like this

set LIVEBOOK_CACERTFILE =C:\some\path

System.get_env(“LIVEBOOK_CACERTFILE”)

The LIVEBOOK_* env vars are not passed to the runtime, so you won’t see it in the notebook : )

Is the error the same after setting this env var?

jonatanklosko · November 7, 2024, 10:58am

Actually, we don’t respect HTTP_PROXY at the moment, I will add that soon.

larshei · November 7, 2024, 11:10am

No luck unfortunately.

From livebook.ex in the config_runtime function, it seems the LIVEBOOK_CACERTFILE should be stored in the application config, but I cannot see that being available either:

jonatanklosko · November 7, 2024, 11:52am

but I cannot see that being available either:

The notebook code executes in a separate, clean Elixir node, so it’s not applicable there.

I will add the proxy support shortly and let you know.

larshei · November 7, 2024, 11:57am

thanks a lot!

jonatanklosko · November 8, 2024, 8:47am

@larshei I’ve just merged the change (Support http proxy configuration via env variables by jonatanklosko · Pull Request #2850 · livebook-dev/livebook · GitHub) and built new nightly release. You can download it here. Livebook should now respect the HTTP_PROXY env var. Let me know if it works, or if there’s anything missing still.

larshei · November 8, 2024, 1:15pm

Thanks for being so responsive and the quick PR & merge.

In livebook built from the current main branch,
I can see the configuration is set as expected:

> $env:LIVEBOOK_CACERTFILE
C:\internal_root_ca.pem
> $env:HTTP_PROXY
http://proxy.konzern.intern:8080
> $env:HTTPS_PROXY
http://proxy.konzern.intern:8080

> iex.bat -S mix phx.server

1> "https://google.com" |> URI.parse |> Livebook.Utils.mint_connect_options_for_uri()    
[
  proxy: {:http, "proxy.konzern.intern", 8080, []},
  transport_opts: [cacertfile: "C:\\internal_root_ca.pem"]
]

However, I can only reach webpages hosted in our network internally.
Web pages that are not hosted on the internal network cannot be reached, regardless of whether the traffic is routed through the proxy or exempted from it.

This is probably an issue of me not understanding the proxy configuration in the company well enough, so I’ll have to dig into that.

Thanks for your quick replies and the livebook update!

jonatanklosko · November 12, 2024, 5:25am

However, I can only reach webpages hosted in our network internally.

Do you mean a Livebook feature, such as loading notebooks from URL, or do you mean programmatic requests from the notebook code?

larshei · November 13, 2024, 8:18am

Havent tried loading a notebook yet - programmatic requests and requests from livebook itself fail (meaning i cannot search or install any dependencies).

Before, I thought this was the case because of proxy/ca settings not being set, but now that they are I am quite certain this is a todo on our end.

jonatanklosko · November 13, 2024, 8:26am

For installing dependencies, I think you also need to set HEX_CACERTS_PATH, so that Mix.install uses the certs when talking to Hex.

For programmatic requests, such as Req.get, you need to set the proxy options yourself, as if you were writing a regular Elixir app/script.

Searching for dependencies should work though, since it’s Livebook making the requests. Is it still the same error?

larshei · November 13, 2024, 12:08pm

HEX_CACERTS_PATH is set:

> $env:HEX_CACERTS_PATH
C:\internal_root_ca.pem

Pulling dependencies from a mix project works as intendend:

> mix phx.new hello_web
* creating hello_web/lib/hello_web/application.ex
* creating hello_web/lib/hello_web.ex
* creating hello_web/lib/hello_web_web/controllers/error_json.ex
...
Fetch and install dependencies? [Yn] y
* running mix deps.get
* running mix assets.setup
* running mix deps.compile

We are almost there! The following steps are missing:
...

so does mix hex.info:

 mix hex.info ecto
A toolkit for data mapping and language integrated query for Elixir

Config: {:ecto, "~> 3.12"}
Locked version: 3.12.4
...

Finding depencies through livebooks search

Results in an error message.

Failed to make a request, reason: 
{:failed_connect, [{:to_address, {~c"hex.pm", 443}}, {:tls, [
server_name_indication: ~c"hex.pm",
cacertfile: ~c"\"C:\\internal_root_ca.pem\"",
verify: :verify_peer,
customize_hostname_check: [
  match_fun: #Function<6.75820660/2 in :public_key.pkix_verify_hostname_match_fun/1>]],
  {:options, {
     :cacertfile, ~c"c:/Users/me/AppData/Local/Livebook/\"C:/internal_root_ca.pem\"", 
{:error, :enoent}}}}]}

Looks like building the path to the certfile assumes a relative path and also adds " as escaped characters.

Installing depencies with Mix.install

Instead of adding a dependency through search, I added it “manually” and tried to start the runtime:

Mix.install([
  {:ecto, "> 0.0.0"}
])

Leads to

13:03:25.702 [error] beam\beam_load.c(607): Error loading function 'Elixir.Hex.Netrc.Cache':fetch/1: op make_fun2 u:
  no specific operation found



13:03:25.710 [error] beam\beam_load.c(607): Error loading function 'Elixir.Hex.Netrc.Cache':fetch/1: op make_fun2 u:
  no specific operation found



13:03:25.712 [error] Writer crashed (:"The pipe is being closed.\r\n")

13:03:25.721 [notice] Application hex exited: exited in: Hex.Application.start(:normal, [])
    ** (EXIT) an exception was raised:
        ** (ArgumentError) The module Hex.Netrc.Cache was given as a child to a supervisor but it does not exist
            (elixir 1.17.2) lib/supervisor.ex:797: Supervisor.init_child/1
            (elixir 1.17.2) lib/enum.ex:1703: Enum."-map/2-lists^map/1-1-"/2
            (elixir 1.17.2) lib/supervisor.ex:783: Supervisor.init/2
            (elixir 1.17.2) lib/supervisor.ex:707: Supervisor.start_link/2
            (kernel 10.0) application_master.erl:295: :application_master.start_it_old/4

13:03:25.731 [notice] Application inets exited: :stopped

13:03:25.718 [info] init got unexpected: {:io_request, #PID<0.92.0>, #Reference<0.2950530402.328466435.66149>,
 {:put_chars, :unicode,
  "Failed to write log message to stdout, trying stderr\n"}}

13:03:25.738 [notice] Application ssl exited: :stopped

13:03:25.738 [notice] Application public_key exited: :stopped

13:03:25.738 [notice] Application asn1 exited: :stopped

13:03:25.738 [notice] Application crypto exited: :stopped
Could not start Hex. Try fetching a new version with "mix local.hex" or uninstalling it with "mix archive.uninstall hex.ez"

As suggested, I ran mix local.hex (successfully) and then restarted the livebook application.

jonatanklosko · November 13, 2024, 1:44pm

cacertfile: ~c"\"C:\\internal_root_ca.pem\"",

Yeah this looks wrong. If you open iex and do System.get_env("LIVEBOOK_CACERTFILE") does it appear with the quotes or without?

larshei · November 13, 2024, 3:23pm

This looks fine:

1> System.get_env("LIVEBOOK_CACERTFILE")
"C:\\internal_root_ca.pem"

jonatanklosko · November 13, 2024, 3:34pm

Just to double check, do you have %USERPROFILE%\.livebookdesktop.bat or did you set it globally. If the file is there, are there quotes or no?

larshei · November 13, 2024, 3:45pm

File is present.

Content was

set LIVEBOOK_CACERTFILE="C:\internal_root_ca.pem"

and is now

set LIVEBOOK_CACERTFILE=C:\internal_root_ca.pem

which leads to

>  System.get_env("LIVEBOOK_CACERTFILE")
"C:\\internal_root_ca.pem"

The env variable is also defined in the user environment variables in windows’ advanced system settings as C:\internal_root_ca.pem - which is used in this case?

jonatanklosko · November 13, 2024, 3:47pm

System.get_env(“LIVEBOOK_CACERTFILE”)

When you do it in IEx it uses the global one, not the one in .livebookdesktop.bat.

So maybe the quotes were the issue? Does the package search work now?

which is used in this case?

I think .livebookdesktop.bat should take precedence.

larshei · November 13, 2024, 3:58pm

search works now! Thanks, well debugged I thought the quotes in the file make no difference, I wouldnt have caught this.

However, installing dependencies did not work yet, error remains:

error] beam\beam_load.c(607): Error loading function 'Elixir.Hex.Netrc.Cache':fetch/1: op make_fun2 u:
  no specific operation found

Could be a version mismatch between Elixir and OTP?
Can I see what versions are used in the installed livebook version?