Bert.js - Binary ERlang Term serialization library for Javascript (with Elixir support)

@dgmcguire Benchmark it. ETF serialization is not as fast as you’d think, even JSON can be faster at times (though less expressive).

5 Likes

Yup, I’ve learned to trust nothing but benchmarks and to not trust them either :joy:

2 Likes

Bert.js with your AST diffing (is that iolistt?), they could work with Phoenix.LiveView. That would be interesting.

I’m not really sure this is a good idea: using :erlang.binary_to_term() may lead to atoms exhaustion.
A malicious user might craft a payload that contains a huge number of new atoms, with just few malicious payloads this might happen: “Atoms are not garbage-collected. Once an atom is created, it is never removed. The emulator terminates if the limit for the number of atoms (1,048,576 by default) is reached.”. In my opinion this sounds like a denial of service.

I think that JSON might be enough for your case. Also, like I said it before, don’t diff AST. Diff a special internediate representation.

I think bert-encoded-html over websocket has been already done by n2o project a few years ago. I don’t remember it using any kind of diffing back then, though.

@alexdovzhanyn you might be interested in reading through the sources of some other bert encoders/decoders in js like https://github.com/synrc/n2o/blob/master/priv/protocols/bert.js and https://github.com/discordapp/erlpack, in case you haven’t already.

:erlang.binary_to_term/2 has an option to prevent DoS via the atoms table

3 Likes

I still need to go back and reread all of your advice, but I’m wondering if you have a different idea of what I mean by AST. I’m not talking about diffing elixir AST. I’m diffing html AST

(and really it’s not even a tree diff right now, like I said about using floki at runtime - right now I’m focusing on a usable library before an optimized library, but I am getting very close to moving to optimizations)

when you say intermediate representation do you mean the flattened out nodes list?

We are talking about the same thing then. Sorry.

I think decoding functions would also create atoms.

The safe version of the decoder also prevents that.

2 Likes

You can pass an option to :erlang.binary_to_term/2 to tell it not to create new atoms (will error if someone tries). (EDIT: as @dgmcguire says further below that post)

1 Like

Thanks, good point :slight_smile:
I made a pull request to Bert.js, https://github.com/ElixiumNetwork/bert-elixir/pull/1 to make it clear
I think we should always keep Elixir comunity security aware :slight_smile:

1 Like

There is a discussion on a related topic here: Anyone using Erlang External Term Format (ETF) instead of e.g. JSON?, so I posted there the link I was going to post here.

3 Likes

Hey, tanks for mention our library!
Just wanted to notice that we updated bert.js URL in master:
https://github.com/synrc/n2o/blob/master/priv/bert.js
and I also wrote some comments on our implementation:
https://ws.n2o.space/man/bert.js.htm

3 Likes

@5HT Your username is like CRAZY familiar… what do you do? Did you write nitrogen or it’s n2o fork or something?

Hello, OvermindDL1! The original Nitrogen author (as you may already know) is Rusty Klophaus, we picked up nitrogen_core in late 2013 and forked it to the N2O project which is one of carefully selected, frozen and adopted erlang packages n2o.space. Since original n2o announcement at HN, we added support of MQTT and did 21 releases of n2o.

2 Likes

Very cool!

Question time: When is it useful to use Bert.js instead of JSON? (Because while BERT/ETF is Elixir’s ‘native’ serialization, JSON is the browser’s native serialization, so I’m not sure about the advantages of using BERT here.) How does its JS (de)serialization compare in sense of execution speed/memory usage vs JSON?

1 Like

I think the logic is that the server is a more limited resource, so having a fasted serialization library there becomes more important. Although I haven’t had any problems with encoding speed of json in my own apps yet.

1 Like

I would say that using BSON, MessagePack, bert.js, JSON alone without generators is a naїve approach. The more mature approach is to use parser generators, like ASN.1, protobuf, synrc/bert. The latter one provides gRPC/protobuf, JavaScript, and Swift SDK generation from HRL files. But even in the naїve case, bert.js provides type hints immediately after DataView unpack. Also, that (type hints) are useful for client-side validation, and for sure in that way we remove unnecessary abstraction layer at the server side. Harder, Better, Faster, Stronger!

3 Likes