ElixirLS cannot find Elixir sources (ASDF, pre-compiled Elixir)

I can’t believe I am the only one having this issue, but I wasn’t able to find a solution

I used ASDF to install Elixir and Erlang and now I added vim-lsp and vim-lsp-settings to use ElixirLS, but when starting ElixirLS, I get an error that it could not find the elixir sources:

{
  "jsonrpc":"2.0",
  "method":"window/logMessage",
  "params":{
    "message":"Elixir sources not found (checking in /home/runner/work/elixir/elixir). Code navigation to Elixir modules disabled.",
    "type":3
  }
}

It sources the elixir sources in /home/runner/work/elixir/elixir but that directory does not exist. That path comes from Enum.module_info()[:compile][:source] which ElixirLS uses to find the elixir sources:

iex> Enum.module_info()[:compile][:source]
~c"/home/runner/work/elixir/elixir/lib/elixir/lib/enum.ex"

I think it is because ASDF installs pre-compiled Elixir

Is it somehow possible to use ElixirLS with pre-compiled Elixir? Or is there anything I need to do differently?

I found Issues with ElixirLS: Elixir sources not found and error while running own Elixir LS Version - #2 by jngcorvus but there someone just cloned the elixir sources to that directory, but this feels wrong.