Bert.js - Binary ERlang Term serialization library for Javascript (with Elixir support)

Bert.js is a javascript library that allows (de)serialization of Erlang terms between javascript and Erlang/Elixir.

Here it is on GitHub and on NPM

13 Likes

Looks useful though you should detail which of the ETF types you donā€™t support (like closures and so forth) and of the ones that you do support then how they map to javascript, for documentation reasons. :slight_smile:

3 Likes

Thanks! Iā€™m still in the process of writing up documentation, but Iā€™ll update this thread once I have that done :relaxed: :relaxed:

3 Likes

I was just looking at this the other day wondering if there were any libraries doing active development on this. I canā€™t wait to see how this works with my library thatā€™s diffā€™ing AST (HTML parsed into AST) and pushing patches over the wire. I imagine berts will be quite the optimization

4 Likes

Let me know how that turns out! Iā€™m curious as well

1 Like

@dgmcguire Benchmark it. ETF serialization is not as fast as youā€™d think, even JSON can be faster at times (though less expressive).

5 Likes

Yup, Iā€™ve learned to trust nothing but benchmarks and to not trust them either :joy:

2 Likes

Bert.js with your AST diffing (is that iolistt?), they could work with Phoenix.LiveView. That would be interesting.

Iā€™m not really sure this is a good idea: using :erlang.binary_to_term() may lead to atoms exhaustion.
A malicious user might craft a payload that contains a huge number of new atoms, with just few malicious payloads this might happen: ā€œAtoms are not garbage-collected. Once an atom is created, it is never removed. The emulator terminates if the limit for the number of atoms (1,048,576 by default) is reached.ā€. In my opinion this sounds like a denial of service.

I think that JSON might be enough for your case. Also, like I said it before, donā€™t diff AST. Diff a special internediate representation.

I think bert-encoded-html over websocket has been already done by n2o project a few years ago. I donā€™t remember it using any kind of diffing back then, though.

@alexdovzhanyn you might be interested in reading through the sources of some other bert encoders/decoders in js like https://github.com/synrc/n2o/blob/master/priv/protocols/bert.js and GitHub - discord/erlpack: High Performance Erlang Term Format Packer, in case you havenā€™t already.

:erlang.binary_to_term/2 has an option to prevent DoS via the atoms table

3 Likes

I still need to go back and reread all of your advice, but Iā€™m wondering if you have a different idea of what I mean by AST. Iā€™m not talking about diffing elixir AST. Iā€™m diffing html AST

(and really itā€™s not even a tree diff right now, like I said about using floki at runtime - right now Iā€™m focusing on a usable library before an optimized library, but I am getting very close to moving to optimizations)

when you say intermediate representation do you mean the flattened out nodes list?

We are talking about the same thing then. Sorry.

I think decoding functions would also create atoms.

The safe version of the decoder also prevents that.

2 Likes

You can pass an option to :erlang.binary_to_term/2 to tell it not to create new atoms (will error if someone tries). (EDIT: as @dgmcguire says further below that post)

1 Like

Thanks, good point :slight_smile:
I made a pull request to Bert.js, https://github.com/ElixiumNetwork/bert-elixir/pull/1 to make it clear
I think we should always keep Elixir comunity security aware :slight_smile:

1 Like

There is a discussion on a related topic here: Anyone using Erlang External Term Format (ETF) instead of e.g. JSON?, so I posted there the link I was going to post here.

3 Likes

Very cool!

Question time: When is it useful to use Bert.js instead of JSON? (Because while BERT/ETF is Elixirā€™s ā€˜nativeā€™ serialization, JSON is the browserā€™s native serialization, so Iā€™m not sure about the advantages of using BERT here.) How does its JS (de)serialization compare in sense of execution speed/memory usage vs JSON?

1 Like