Qqwy

Qqwy

TypeCheck Core Team

Anyone using Erlang External Term Format (ETF) instead of e.g. JSON?

So for Planga, we are experimenting with communicating between nodes (written in different languages; in our case we currently have an Elixir and a Ruby node).

This communication is now using RabbitMQ, with Erlang’s ETF (External Term Format) as serialization format, rather than JSON. We chose this because:

  • It gets minified automatically. (With JSON you can hope for gzipping, but I don’t know if e.g. RabbitMQ will do so)
  • There are more datatypes that can be directly encoded (JSON does not differ between integers and floats, all numbers are only guaranteed to have limited IEEE754 precision.)
  • There are no restrictions in top-level types (in JSON the top-level has to be either an object or an array).

Specifically, between Ruby and Elixir, we’re able to simply send big-number integers over the wire, as well as e.g. symbols/atoms.

However, there obviously are also drawbacks:

  • Be mindful that your symbol table might be filled (memory leak === DoS potential) if reading untrusted data.
  • IIRC ETF could contain e.g. encoded functions, which you do not want to execute if they are from an untrusted source (And ‘stored procedures’ are things that are of questionable usefullness according to many developers).

So I’m wondering: Are there other people that like using ETF over JSON or another format? Why or why not?

First 10 of 11 Posts Switch mode

tty

tty

We currently use BERT between Erlang/Java in one subsystem and JSON between Erlang/Java in another. I have also use BERT in other projects.

We found using JSON to be a PITA to change and update in comparison to BERT. This is partly because, at the very least, the Java side we can rely on (minimal) compiler typechecking.

Although you could use a JSON validator the Erlang validator was feature weak (4 years ago). YMMV.

I prefer BERT because it is a wire protocol and compact. I would even consider protobuf over JSON/XML any day.

wojtekmach

wojtekmach

Hex Core Team

Hex.pm API responds with JSON or ETF. The latter is very convenient as we don’t need a JSON parser to understand it:

curl --silent -H "accept: application/vnd.hex+erlang" https://hex.pm/api | elixir -e "IO.read(:stdio, :all) |> :erlang.binary_to_term() |> IO.inspect()"
%{
  "documentation_url" => "http://docs.hexpm.apiary.io",
  "key_url" => "https://hex.pm/api/keys/{name}",
  "keys_url" => "https://hex.pm/api/keys",
  "package_owners_url" => "https://hex.pm/api/packages/{name}/owners",
  "package_release_url" => "https://hex.pm/api/packages/{name}/releases/{version}",
  "package_url" => "https://hex.pm/api/packages/{name}",
  "packages_url" => "https://hex.pm/api/packages"
}
11
Post #2
Eiji

Eiji

@Qqwy I’m using BERT as often as possible. JSON is used only if it’s requried. I love to use such data format, because I do not need to add any external library to write full work result with minimal number of lines + all what you have already said. For me main use case is for Elixir only (my private projects), temporary with JavaScript (client) and again Elixir only (scenic client + Elixir to WebAssembly client) in future.

rvirding

rvirding

Creator of Erlang

Have you tried using :erlang.binary_to_term/2 which has a safe option which is designed to help with this. binary_to_term/2

10
Post #4
Qqwy

Qqwy OP

TypeCheck Core Team

Wow! Great! I was not aware of this feature, and I will immediately start using it! :smiley:

michalmuskala

michalmuskala

I wish there was another one called data_only or something similar that would forbid funs. If that was the case, it probably would cover all the needs automatically. Right now, in most cases, I need to traverse the decoded data to check if there were any funs in there.

Are you using BERT as defined in http://bert-rpc.org/ or :erlang.term_to_binary directly? They are two different things. In particular BERT itself doesn’t do maps and has a separate, much more verbose encoding.

Eiji

Eiji

Oh, sorry I though that :erlang.term_to_binary is BERT implementation. Is there any other implementation written in Erlang? If so then which one is better for Elixir ↔ other language and ElixirElixir?

tme_317

tme_317

Any downside to using Plug.Crypto.safe_binary_to_term to filter out any funs? At least that’s what I’ve been doing… and it looks like they have recently extracted plug_crypto to a separate library in case you’re not already using plug itself.

Since options are passed to erlang I guess you could use Plug.Crypto.safe_binary_to_term(bin, [:safe]) to protect against atom exhaustion also?

michalmuskala

michalmuskala

No downside, just that it’s slower than a native option could be since you have to traverse the data after it’s decoded. For now that’s the best option, though.

bettio

bettio

Yesterday I wrote a post on this topic, so I would like to share my opinion here https://blog.ispirata.com/how-to-destroy-your-application-using-erlang-binary-to-term-1-575ff7d05333 (I was going to talk about this on the Bert.js topic). Let me know if you don’t agree :slight_smile:

Where Next?

Trending in Discussions Top

AstonJ
As the title says, please share what you’ve been up to with Elixir. Whether that’s been learning it, looking into it, making stuff with i...
2977 91561 914
New
byu
@chrismccord : I just saw the Extract AGENTS.md from Phoenix.new into phx.new generator commit to the phoenix project. My initial shotgu...
New
arcanemachine
I was working on an Ecto migration and I needed a timestamp. So, for the nth time, I looked up the different data types for timestamps, a...
New
AstonJ
Just a general thread to post chat/news/info relating to AI/ML stuff that may be relevant for Nx now or in the future. Got anything to sh...
New
type1fool
I just stumbled on a newly redesigned elixir-lang.org. :tada: It looks like @Software_Mansion did the work, and I think it is generally a...
New
juhalehtonen
There has been a thread to discuss the Stack Overflow Developer Survey on this forum every year since 2018, so here’s yet another one for...
New
alexslade
Fly’s CEO posted this recently - Turn And Face The Strange · The Fly Blog It says that Fly is going all-in on sprites, which is a worry ...
New

Other Trending Topics Top

JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New
ausimian
Emily is an Elixir library that runs Nx computations on Apple’s MLX. Install it as the default Nx backend and Nx, defn, Axon, Nx.Serving,...
New
akoutmos
@hugobarauna and I (Alex Koutmos) have been hard at work on writing a book on Nerves that takes you from simply blinking LEDs to building...
New
bjorng
We want to introduce a new native datatype to Erlang: native records. Although replacing all tuple records with native records is not our...
New

We're in Beta

About us Mission Statement