What do your ideal versions of Erlang/OTP/BEAM look like?

One thing I love about our community is the high standard set by the Elixir, Phoenix, Nerves and Erlang core teams… another thing I like is how others wish to raise those standards even higher :lol:

So I thought it would be a nice idea to post a series of threads for just that :003:

Please let us know what your ideal versions of Erlang/OTP/BEAM look like in this thread :smiley:


Other threads in this series

2 Likes

Remove most of the built-in applications to Hex libraries, namely:

  • xmerl
  • wx - however that would be hard due to observer
  • tftp
  • snmp
  • megaco
  • ftp
  • ssh
  • diameter
  • os_mon

I am not sure about ssl and crypto modules though.

@hauleth I believe that most of that is coming with future OTP releases but remember, if you generate your own elixir application releases, e.g. with distillery, you’ll only get what you need anyway.

Of the list above, I’d speculate that xmerl, snmp, ssh, and os_mon would remain and the rest is up for grabs.

That said, there are a handful of things I’d like to see in “Future OTP”:

  • fully bare metal BEAM I’d love to have, see cloudozer’s xerg https://www.slideshare.net/vsovietov/erlang-dnipro-2014-erlangonxen the demo is sadly no longer there but being able to ship a fully operational death star / BEAM without OS overhead to a typical hypervisor would be awesome
  • a secure boundary for distributed erlang, right now you can encrypt the channel with TLS but it’s still a free-for-all once you brute force the cookie, it would be great to restrict that a bit
  • easier way of injecting secrets into the BEAM at runtime. I’m not really happy with any of the solutions yet, and I’d like something simpler to start with. The much reviled {:system, VALUE} is probably 80% sufficient for most peoples’ use cases. I appreciate the need to move carefully and slowly on this though, once you commit to something you can never take it out.
  • some different approaches to distributed erlang to allow getting past the ~ 70 node limit. With care and tuning you can “sneak” past this, but it starts to get dicey. Personally I’ve never needed this but I wants it. SD Erlang looked really promising and then it disappeared from the horizon.
  • I’d like to see the 2G DETS limit nuked from orbit, and have some sort of high-speed lock-free integrated datastore shipped as part of OTP. eleveldb is I believe widely** used, e.g. at Klarna but it’s not part of “normal” OTP
  • more interesting uses of NIFs in cross-platform type-safe languages
  • named function parameters a la ocaml type punning available in the compiler
  • a high-performance HTTP client (cue Mint ?) integrated into either Elixir core, or Erlang/OTP - httpc has flaws, many forks, but nothing folded back into it
  • more people using this IMO highly under-rated ecosystem elixir+BEAM is awesome
3 Likes

Yes, of course. But the main reason why having such utilities outside main tree is that these can evolve faster if these are external dependencies. Only that is huge and just for that reason it would be worth splitting.

I am aware of it, it was named Erlang-on-XEN and then renamed to LING just before the project died. Fortunately there are now few other projects that provide unikernel or unikernely workflow:

  • rebar3_osv plugin - which provides Erlang runtime in OSv
  • Nerves - unikernel-like build for Elixir projects
  • AtomVM - while not unikernel it is small implementation of subset of Erlang to run on small and embedded devices, can be used as a base for unikernel
  • Enigma - another reimplementation of Erlang runtime, together with projects like libhermit it can be used as a base for unikernel as well

So we are slowly getting there, which makes me happy as I see Erlang as a perfect solution for unikernel runtime (as Erlang VM is basically an full soft-RTOS).

Cookies was never (at least since open sourcing of Erlang) meant as a security solution, it is more “simple check” that you connect to correct cluster. And with TLS you can restrict connections only to clients that use proper certificate, so you can use that as a security measure. Not perfect, but it is there. Once upon a time I have an idea for replacing EPMD (the main insecure component of distributed Erlang) with Consul service discovery or something like that. Maybe one day.

I think that current solutions with:

  • config providers (in Distillery) and init/start functions in Ecto, Phoenix, Application are good enough, what these solutions need though are the broader knowledge of them and broader support. {:system, env_name} was terrible hack that should never happen.

I would say that httpc is “good enough” and it do not need to be replaced much. The core should provide the simplest, easiest, and smallest possible solution that works, and if you want heavy lifting then go for external dependency (for exactly the same reasons I have mentioned above). Just leave httpc as is and go for Mint/Gun/anything other when you need something more complex.

1 Like

For the future OTP, I imagine this (my personal opinion):

  1. Modernise supervision system, there are already libraries on that topic: director, supervisor2, supervisor3, gen_tracker - so would be nice rethink supervisors (using experience from this libraries) to create or update existing.
    I personally like approach of director and gen_tracker with additional callbacks (like handle_exit and after_terminate), which gives great flexibility. So, introduce behaviour supervised would be great in this context, which use callbacks for handle_exit, and so on, so that for example it is easy for an user to add own backoff strategy directly to supervision.

Actually, Elixir already introduced this behaviour, without naming it! Check child_spec, it is supervised behaviour callback already.

  1. Distribution: There is already improvement for large chunks of data in 22 Erlang, may be can be there pooled connections or like have an option for a send to use either control or data channel? (Like it was implemented in riak, for control channel erlang distributed was used, for data tcp connection). So, that actually build-in distribution even for data intensive load will be good enough.

  2. High level distribution: so, there are many occurring tasks in distribution, share data between nodes (distributed ets), find process (distributed registry), start only one instance of a process in a cluster and if node goes down, restart it on another node (which is than findable with distributed registry). Would be nice to have de-facto an industry standard raft protocol for consensus-based strong consistency guaranty primitive to implement distributed cache, registries and so on for distributed applications. There, actually, it can be heavily inspired by etcd/raft design, as it used in many distributed systems and databases, like dgraph, cockroachdb and so on. In Erlang/Elixir there were many approaches to solve this problem: riak_ensemble, rabbitmq/ra, rafted_value, but no one of this solution is extensible enough(as I know) like a Go alternative. It would be nice to have a stable battle-tested ground(like etcd/raft) which can be used for building high level distributed applications. This can replace global with a strong consistent solution and so on.

  3. Single file executable and release building should be inside of OTP, probably: supporting single file executable can simplify adaptation of Erlang/OTP. There are again different people tryed different solutions, most notable is mad (beam bundles). It would be nice to have this out of the box with possibility to include own nifs. It will simplify even building desktop application, what could make Erlang/OTP easier to use as general purpose language. Unify release handling with pluggable interactive shell and rpc evaluation will make it easier to replace: relx, distillery, Elixir own next version release handling. Because, they are doing actually to 99% the same thing.

  4. Persistence: it is less important, as other 4 points, so it is still nice to have: rebuild mnesia on top raft, so that it will be good enough for simple distributed application, but that still can survive in strong consistent manner(predictable output) networks splits. It is not so important, because with availability of another databases(cockroachdb, even mongodb since v4.0) and possibility to build DSL(ecto and generally macros in Elixir) for this database, there is no more real need for this. But for having bundled experience again(install Erlang and you can build easily application without any extra dependencies), as why mnesia was there - it will be really nice. Transactional, distributed database, which survives networks splits in consistent way - this will be a really great feature for bundled experience.

  5. Elixir Logger based on logger. So, probably logger at least needs design change, to replace evaluation to allow plugging Elixir printing protocols. But, having unified mechanism for logging will profit better integration between Erlang and Elixir(and other future languages) libraries and applications.

I have touched topics, where are mostly some solutions exists(and different people had the same problem), but if there will be a solution with extensible design and as a part of OTP, so many different people can profit from it.

3 Likes