Good question, Ryan. It exposed a real weakness that needed fixing.

Short answer: the allowlist would not have caught a LiteLLM-style attack on an allowlisted package. But thanks to your question, as of commit 00d87ad, Vet now automatically diffs every dependency against its previous version on Hex, and those findings bypass the allowlist entirely.

How it works: Hex keeps every published version permanently. When Vet scans your lock file, for each dependency it fetches the previous version from Hex and compares the two. If the version transition introduced new dangerous patterns (compile-time env access, network exfiltration, new file categories), those get flagged as [VERSION DIFF] findings. These findings are not subject to the allowlist. The allowlist says “we reviewed this package’s existing behavior.” The version diff says “the behavior changed.”

Your thought experiment: If litellm were allowlisted and version 1.82.8 was compromised, Vet would fetch 1.82.7 from Hex, diff the two, detect the new @before_compilebefore_compilebefore_compilebefore_compile hook reading AWS_SECRET_ACCESS_KEY and POSTing it, and flag it as a CRITICAL profile shift. The user never needed to have 1.82.7 installed. Hex has it.

This works for first-time installs too. If you install litellm for the first time at 1.82.8, Vet still diffs against 1.82.7 (fetched from Hex) and catches the transition.

Vet also runs a lookback diff (3c616df), comparing the current version against the version from 10 releases ago (or the earliest available version if the package has fewer than 10 releases). This catches gradual introduction of malicious code across multiple small versions where no single step looks suspicious but the aggregate does.

Your earlier feedback about the aliases bug and the Phoenix false positives already led to two significant fixes. This question led to two more. The project is measurably better because you tried it on a real project and asked hard questions. You rock.

Pull main and try mix vet on your project. You should see [VERSION DIFF] entries for any dependency where the version transition looks unusual. mix vet --no-diff disables it if you want faster scans.

Thanks for actually trying it! You found two real bugs.

The aliases-as-packages issue: the AST walker was scanning the entire mix.exs looking for {atom, _} 2-tuples, which happily caught keyword pairs from your aliases/0 function (setup:, precommit:, "ecto.setup":). Those then got passed to mix vet.check, which looked them up on hex.pm, found nothing, and reported them as CRITICAL phantom packages. Embarrassing.

Fixed by scoping extraction to the body of the deps/0 function only. The new version also handles 3-tuple deps with options like {:phoenix_live_view, "~> 1.0", only: :dev} (which in AST is {:{}, meta, [name, version, opts]} — different shape from a plain 2-tuple, easy to miss).

“Returns quite some errors”: the built-in allowlist only covered ~20 packages, but mix phx.new --database postgres pulls in ~30 deps, most of which legitimately trip Vet’s checks — bandit calls the network because it is the network, esbuild runs system commands, gettext reads files at compile time, plug_crypto uses :crypto. So you got a wall of false positives. Added ~50 entries covering the Phoenix 1.7+ ecosystem (bandit, thousand_island, phoenix_pubsub/html/live_view/live_dashboard/live_reload/ecto/template, postgrex, db_connection, telemetry_metrics/poller, gettext, swoosh, esbuild, tailwind, dart_sass, dns_cluster, plug_crypto, castore, nimble_pool, floki, websock_adapter, bcrypt_elixir, …).

Both fixes are in db64127. Added a regression test using a Phoenix-shaped mix.exs with both deps and aliases, plus a property test that generates random mix.exs files with both and proves no alias name leaks. Full suite is 62 properties + 457 tests, all green.

Pull main and try again — if anything is still noisy on your project I’d genuinely like to know. Real test cases beat synthetic ones every time.

I have some:

• :ssh and :ssh_sftp aren’t checked - I can freely open some connection to remote server or open remote shell
• :ftp is not protected, so I can download or upload any data I want
• :httpd is not checked, so the second “hidden” HTTP server can be ran
• I can use :prim_file module - while it isn’t public API, I think that attacker will not give a damn about breaking API (and once upon a time I needed genuinely use that module, because “stable” API do not provide such functionality)
• I can run any external executable using Mix.shell().cmd(…) (which is by the way an approach that should be used in Mix tasks)
• :erlang.open_port is free to be used
• I can use Tesla or hackney or other HTTP client just fine, the same with sockets where I can use procket library to open any unsafe socket
• I can use defdelegate to circumvent any function call check

  defmodule TestMod do
    defdelegate foo(str), to: String, as: :to_atom
  
    def convert(str) do
      foo(str)
    end
  end
  

• While I cannot use apply I can still write function like def call(mod), do: mod.connect(~c"i.will.not.to.anything.promise.dev", []) and be able to connect to remote server
• I can use def "$handle_undefined_function"(name, args) to do more nasty things
• I can use :inet_res functions to leak data by sending prepared DNS requests containing private data
• I can use :epp.scan_file to read file content (it will be mangled, but in many cases it may be possible to extract required data from there

I appreciate your effort, the problem I see is that Erlang VM is just unsafe VM and I do not believe that there is a way to do it in a way, that will not create false sense of security. I strongly believe that tool that provide false sense of security is worse option than no tool at all.

What this tool currently provide is more of a sanity check against accidental data leakage than checking against actually malicious party trying to obscure their actions. It may be useful anyway, just not as a defence against malicious library providers (I deliberately do not use term “supply chain”, as open source authors aren’t “supply chain”).

If you want to secure against actively malicious parties, then:

1. You need to think as malicious attacker. If someone wants to hide what they are doing, they for sure will not fall into trap where you can simply use “grep on steroids” to see that it is doing bonkers stuff
2. In my opinion if someone want to hide what they are doing, then without higher level sandbox (either Erlang VM level or OS level) you cannot really catch that. This library can be circumvented in super simple way already - just write some Erlang module in your project, and you are safe to go and do anything, as it scans only Elixir code.