2019 BEAM wish list

[blatyo]

Now that we’re starting a new year, what things would you like to see happen in BEAM land?

My list:

1. telemetry and telemetry_metrics reach a stable API and start to be used widely through the community.
2. Releases starting to make their way into Elixir
3. Work starting on a JIT or some other means for improving speed in the BEAM, where the things that are “too slow” on the BEAM no longer are. I’m thinking specifically of linear algebra and machine learning workloads.
4. More mathematical and machine learning libraries making their way to the BEAM (Probably as NIFs). Model runners are a good start, but it would be nice to be able to create the models on the BEAM as well.
5. A comprehensive internationalization and localization library that uses ICU standards. I’ve been particularly impressed with Intl, which is making its way into browsers.

[victorolinasc]

Nice list there! Here’s my complementary:

• I’d like to see more work on distribution. Some ideas from partisan could/should be applied to OTP itself like different channels for gossiping and communication in between the cluster and an easier way to change topologies. Maybe more work on the Firenest project could help here too. It seems that the Kademlia work on OTP was stopped (Zandra seems to have left the OTP team) and that is a pitty…
• it’d be interesting to see some Riak like cluster/database/orchestrator appear in pure Elixir just because I could contribute and see the code better than in current Erlang (this very selfish, I know…)
• I STILL think that the language server implementation and associated projects (like elixir_sense) should be handled by the core team much like the Erlang/Swift/Rust/etc… languages are doing. I know it is a matter of manpower but this seems to me one of the reasons why other languages are being successful too (surely not THE reason itself…). I thisnk this plays a role on the success story of other languages mainly because of developer happiness overall.
• I wish for the Emacs support on the Elixir platform to evolve a lot better than 2018 Will try to help here… Still coming to terms with Elisp though…
• Other than that, I hope more cloud solutions (mainly PaaS and not IaaS of course) start seeing the different approaches the BEAM provides like Gigalixir. The “stateless” non-clustered world with lots of layers is, in my opinion, not the best world for the BEAM.

Besides making a wish list I’d like to once again thank the core team and the community for a great 2018 and wish you all the best. Let’s wish for technical features, of course, but also for a fun and happy year to all the non-technical aspects of the community/family/friends too.

[l00ker]

I’d like to see a WASM version of the BEAM running in the browser that would allow us to use Elixir and/or Phoenix on the front-end – LiveView Extreme Edition!

2019?

[kip]

@blatyo

A comprehensive internationalization and localization library that uses ICU standards. I’ve been particularly impressed with Intl , which is making its way into browsers.

I would welcome your feedback on what I can do to make ex_cldr and its frriends ex_cldr_numbers; ex_cldr_dates_times; ex_cldr_units; ex_cldr_lists and a great contribution from @Schultzer of ex_cldr_territories better for the Elixir ecosystem. Oh, and ex_money which is also fully localised.

All of these are based upon CLDR which underpins ICU.

[kip]

@victorolinasc

Besides making a wish list I’d like to once again thank the core team and the community for a great 2018 and wish you all the best.

Hear hear fo the grateful thanks to the core teams of Elixir and Phoenix and all of the other people who make this such a rewarding platform with which to work including of course the OTP team. I work in Elixir because I enjoy the language, the BEAM platform, the community and the style of José’s leadership.

Thank you all for a memorable year of progress, and for whatever amazing stuff you cook up in 2019. I think all the suggestions above would be great - lets hope this inspires lots of new contributors!

[blatyo]

@kip That’s great! To be honest, I was ignorant of the link between CLDR and ICU. I think the main thing I’d want is something that ties all that together. Like the ICU message format.

[kip]

Wow, and being focused in CLDR I wasn’t aware of the ICU message format! Got my 2019 project defined now . Thanks for the pointer!

[adkron]

@keathley, and I recently did a Christmas wish list on Elixir Outlaws that goes right along with this. https://elixiroutlaws.com/26

[tmbb]

Have you been able to found a formal grammar for the ICU message format? I’ve only found informal descriptions and a bunch of slightly incomplete implementations. And more important: is there anyone using ICU messages in practice instead of something simpler like gettext?

Another big problem with MessageFormat is that although I can find the informal description for the message format, I can’t find a standardized file format to store translations. Gettext has one, which means that although the .po/.pot files are generated by Elixir itself, they can be edited with some easy-to-use external tools.

I’ve been thinking of a very simple API, consisting of a single (although a little verbose) translate/2 macro such as this:

use SuperTranslator
# This imports the `translate/2` macro.
# You need to `use` the SuperTranslator because my next idea requires
# registering a custom module attribute
message =
  translate("At {time,time} on {date,date}, there was {what} on planet {planet,number,integer}.",
    domain: "war_of_the_starse",
    context: "A famous event that transpired near Alderaan",
    variables: [
      time: Time.utc_now(),
      date: Date.utc_today(),
      what: "a great disturbance in the force",
      planet: 7
    ]
  )

The :domain key is the same as the one in gettext and the :context key helps supplying some context to the translators. The variables are inserted into the string according to ICU’s rules.

Unrelated to the use of ICU vs Gettext or some other standard, I’ve stumbled upon and idea, which could probably be incorporated into Gettext. Currently, Gettext doesn’t let you internationalize strings in dependencies, because it needs to gather strings while the application is compiled, store them in a GenServer (actually an Agent), and then compile the translations. This doesn’t work with dependencies because those have already been compiled when the main application is compiled.

I’ve built a proof of concept which bypasses this: whenever SuperTranslator (just a name I’m using currently) is used, it registers a module attribute as an accumulator, which is set by the translate macro. That attribute is then stored inside a function by a @before_compile hook. This persists the translate calls in a way that can be found by other applications. Then, the calls can be “consolidated”, for example by doing this:

defmodule MyTranslator do
  use SuperTranslator.Translator, default_locale: "..."
end

After this “consolidation” (I’m deliberately hiding the actual technical details of how this works), the translate calls in the dependencies would be translated too.

I could advance into a full library very easily. Even parsing MessageFormat doesn’t seem too hard. The only thing that’s holding me back is the lack of a a file format to store the translations (so that they can be edited by translators)…

EDIT: this page seems to be a good resource for MessageFormat - https://messageformat.github.io/messageformat/page-about

[kip]

@tmbb no, I haven’t found a grammar. Im reasonably well advanced on a design for implementation with pluggable backends with the intent that it might be able to integrate with trans and potentially with .pot files.

Overall, my intention is to follow the gettext approach of a macro implementation that can do message extraction to a backend at compile time and a function to operate at runtime.

Its not totally clear to me that domain and context are critical but I haven’t really thought about that part yet.

I should have a prototype implementation in the next two weeks.

[LostKobrakai]

This sounds like what https://hexdocs.pm/gettext/Gettext.Backend.html#c:dgettext_noop/2 is supposed to allow for.

[tmbb]

@kip

Its not totally clear to me that domain and context are critical

They aren’t. They’re just nice to have. Especially context.

@kiep

Overall, my intention is to follow the gettext approach of a macro implementation that can do message extraction to a backend at compile time and a function to operate at runtime.

Having a Gettext-like backend forbids doesn’t allow you to translate strings in the dependencies. My approch allows you to gather ALL strings from the app and its dependencies and translate them in your “main” app.

For example, you could seamlessly translate Ecto’s strings from your main appliction. Do you understand the difference?

EDIT: the way this works is by accumulating the strings to translate in a module attribute inside the translate macro and then “store” the attribute inside a function.

The translate macro would expand into something like SuperTranslator.Worker.translate(...). That module would have a default dummy implementation independently of the locale, BUT after consolidation, it would be dynamically replaced by a global translation backend for the whole BEAM instance (which would work not only on your apps but on your dependencies too), according to the generated .po files or whatever.

That way, you can have a mix task that iterates over all modules in all applications and generate your .po files.

The disadvantage is that you’re redefining at compile time a module under the SuperTranslator namespace, but that doesn’t sound too bad.

[lessless]

Strong type system

[tangui]

Set as a native data type, so that sets can be pattern matched and used in guards like maps.

[hauleth]

You can use maps for that. This is exactly how MapSet works.

[wojtekmach]

You can use maps for that. This is exactly how MapSet works.

You can pattern match on MapSet indeed, but it’s discouraged as it’s internal structure is opaque.

[tangui]

Exactly, so you end up either doing this or using plain maps with nil values. It’s ok but it’d be nicer with a native Set datatype IMO

[hauleth]

I wasn’t talking about matching on MapSet but implementing it on your own as a wrapper on maps.

@tangui - IMHO it would be needless complication to the VM. Instead you can use MapSet.new to do such, ex.:

defmodule Foo do
  @match MapSet.new([:a])

  def foo(@match), do: "match"
  def foo(_), do: "not match"
end

IO.puts Foo.foo(MapSet.new([:a, :b, :c])) #=> match

Ergonomics isn’t perfect as you cannot call remote functions in function headers, and I do not think that there is easy way to make MapSet.new/1 macro so such construct would be allowed.

[wojtekmach]

@hauleth

@match MapSet.new([:a])
def foo(@match), do: "match"
def foo(_), do: "not match"

clever!

Btw, we can already write a guard that will check if an element is a member of mapset:

defmodule MapSetTest do
  use ExUnit.Case, async: true

  defguard is_mapset_member(mapset, element)
           when :erlang.is_map_key(element, :erlang.map_get(:map, mapset))

  def f(mapset) when is_mapset_member(mapset, :a), do: true
  def f(_), do: false

  test "it works" do
    assert f(MapSet.new([:a])) == true
    assert f(MapSet.new([:b])) == false
  end
end

Of course, the caveat that mapset is opaque still applies!

[blatyo]

@tangui

Set as a native data type, so that sets can be pattern matched and used in guards like maps.

What would you imagine that looking like?

@wojtekmach

defguard is_mapset_member(mapset, element) when :erlang.is_map_key(element, :erlang.map_get(:map, mapset))

I wonder if it might be useful to contribute something like this to Erlang.