gmile
This is just me daydreaming, kindly please let me know if I this is completely delusional… 
Maybe a naive question, but I’ll try my luck: would it be technically feasible to implement distribution of compiled .beam files for Elixir packages, apart from distributing the source code (e.g. what we have now)? Or there’s a good reason, including a technical one, it’s rather not to be done / explored?
The key goal of this would be to reduce app compilation times, especially for big applications.
For example, right now, a particular app at my employer is taking a good 5 minutes to compile from scratch on a not particularly beefy VM on GitHub Actions (there are various strategies to improve the compilation speed there, though that’s not the point of this message).
I am wondering if it’d be feasible to bypass package compilation altogether, by downloading package’s compiled .beam files. This way all .beam files for all packages are downloaded and are ready to be bundled with the application during application compilation step.
Obviously, some packages rely on compile-time configuration options, so it may be next to impossible to compile a package in a one-size-fits-all manner. But maybe there’s a way to ship compiled .beam files for packages that most people could benefit from?
One reason against compiling I can think of is, for example, the potential difficulty of verifying that the compiled package indeed matches the source code, e.g. there’s no malicious code baked in the .beam. OS-level package managers seem to have figured that out somehow though? 
Btw, if this was (or is being) discussed / explored before or elsewhere, I’d be super grateful if you could throw a link at me! 
Where Next?
Trending in Discussions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Phoenix, LiveView, multimedia pipelines, distributed systems. 400 engineers, Elixir since 2017. Available for hire.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ai
- #ecto-query
- #elixirconf-us
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #elixirconf-eu
- #api
- #forms
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 8- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
hst337
Environment defines the compilation, because there are some compile-time inputs like
MIX_ENVor other env variables, or compilation of native dependencies (like NIFs or PortDrivers) is also bound to the environment of the builder.There are plenty of tools which make builds idempotent, isolate the environment, cache builds and define the inputs of the package. I’d suggest using nix
LostKobrakai
This doesn’t invalidate your point at all, but dependencies in a mix project are always compiled with
MIX_ENV=prod, so that one specifically doesn’t change.hst337
Not always, this is configurable via
:envoptionFor example,
{:my_dep, "~> 1.0", env: :test}al2o3cr
There’s some concurrent discussion in this thread, though with a different set of requirements:
One of the “various strategies” is caching the built BEAM files after a
mix deps.compile- if you compute the cache key based just onmix.lock(so that the cache can be used across CI builds that have the same deps) you can get essentially the same results as precompiled packages but without any of the portability / security / etc headaches.gmile
Very nice point, I complete forgot about those kinds of dependencies. Yeah, building packages like that would be problematic
Yeah, we do that and it works mostly fine. This restoration of .beam files from cache is what actually made me think about a “native” way to distribute compiled packages.
hst337
As I said, you can distribute compiled packages using package managers. Some package managers work on all popular platform, like 0install which works on Mac, Linux and Windows or Nix which works on Mac and Linux.
I personally use Nix to build and deploy packages or docker images. Nix is functional where every package is a pure function of it’s inputs and environment. And around this functional idea there is a a smart cache for built packages in a way cache works for pure functions.
Writing Nix packages for Mix applications is pretty straightforward with tools like
mix2nixplus it is officially supported by nixgmile
Another, I believe, obstacle would be the fact that .beam files compiled under a specific version of Erlang are not guaranteed to work on machines running different (likely, previous) versions of Erlang
lukaszsamson
There are some other approaches for distributing beams:
escript mix escript.build — Mix v1.21.0-dev
ez archves (elixir-ls does that in elixir-ls/.github/workflows/release-asset.yml at master · elixir-lsp/elixir-ls · GitHub)