Qqwy
TypeCheck Core Team
Anyone using Erlang External Term Format (ETF) instead of e.g. JSON?
So for Planga, we are experimenting with communicating between nodes (written in different languages; in our case we currently have an Elixir and a Ruby node).
This communication is now using RabbitMQ, with Erlang’s ETF (External Term Format) as serialization format, rather than JSON. We chose this because:
- It gets minified automatically. (With JSON you can hope for gzipping, but I don’t know if e.g. RabbitMQ will do so)
- There are more datatypes that can be directly encoded (JSON does not differ between integers and floats, all numbers are only guaranteed to have limited IEEE754 precision.)
- There are no restrictions in top-level types (in JSON the top-level has to be either an object or an array).
Specifically, between Ruby and Elixir, we’re able to simply send big-number integers over the wire, as well as e.g. symbols/atoms.
However, there obviously are also drawbacks:
- Be mindful that your symbol table might be filled (memory leak === DoS potential) if reading untrusted data.
- IIRC ETF could contain e.g. encoded functions, which you do not want to execute if they are from an untrusted source (And ‘stored procedures’ are things that are of questionable usefullness according to many developers).
So I’m wondering: Are there other people that like using ETF over JSON or another format? Why or why not?
Trending in Discussions
As the title says, please share what you’ve been up to with Elixir. Whether that’s been learning it, looking into it, making stuff with i...
New
@chrismccord : I just saw the Extract AGENTS.md from Phoenix.new into phx.new generator commit to the phoenix project.
My initial shotgu...
New
I was working on an Ecto migration and I needed a timestamp. So, for the nth time, I looked up the different data types for timestamps, a...
New
Just a general thread to post chat/news/info relating to AI/ML stuff that may be relevant for Nx now or in the future. Got anything to sh...
New
I just stumbled on a newly redesigned elixir-lang.org. :tada: It looks like @Software_Mansion did the work, and I think it is generally a...
New
There has been a thread to discuss the Stack Overflow Developer Survey on this forum every year since 2018, so here’s yet another one for...
New
Fly’s CEO posted this recently - Turn And Face The Strange · The Fly Blog
It says that Fly is going all-in on sprites, which is a worry ...
New
Other Trending Topics
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve.
They are GUI (Emerge) and State management (S...
New
Emily is an Elixir library that runs Nx computations on Apple’s MLX. Install it as the default Nx backend and Nx, defn, Axon, Nx.Serving,...
New
@hugobarauna and I (Alex Koutmos) have been hard at work on writing a book on Nerves that takes you from simply blinking LEDs to building...
New
We want to introduce a new native datatype to Erlang: native records. Although replacing all tuple records with native records is not our...
New
Chat & Discussions>Discussions
Latest on Elixir Forum
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #performance











First 10 of 11 Posts
tty
We currently use BERT between Erlang/Java in one subsystem and JSON between Erlang/Java in another. I have also use BERT in other projects.
We found using JSON to be a PITA to change and update in comparison to BERT. This is partly because, at the very least, the Java side we can rely on (minimal) compiler typechecking.
Although you could use a JSON validator the Erlang validator was feature weak (4 years ago). YMMV.
I prefer BERT because it is a wire protocol and compact. I would even consider protobuf over JSON/XML any day.
wojtekmach
Hex.pm API responds with JSON or ETF. The latter is very convenient as we don’t need a JSON parser to understand it:
Eiji
@Qqwy I’m using
BERTas often as possible.JSONis used only if it’s requried. I love to use such data format, because I do not need to add any external library to write full work result with minimal number of lines + all what you have already said. For me main use case is forElixironly (my private projects), temporary withJavaScript(client) and againElixironly (scenicclient +ElixirtoWebAssemblyclient) in future.rvirding
Have you tried using
:erlang.binary_to_term/2which has asafeoption which is designed to help with this. binary_to_term/2Qqwy
Wow! Great! I was not aware of this feature, and I will immediately start using it!
michalmuskala
I wish there was another one called
data_onlyor something similar that would forbid funs. If that was the case, it probably would cover all the needs automatically. Right now, in most cases, I need to traverse the decoded data to check if there were any funs in there.Are you using
BERTas defined in http://bert-rpc.org/ or:erlang.term_to_binarydirectly? They are two different things. In particularBERTitself doesn’t do maps and has a separate, much more verbose encoding.Eiji
Oh, sorry I though that
:erlang.term_to_binaryisBERTimplementation. Is there any other implementation written inErlang? If so then which one is better forElixir↔ other language andElixir↔Elixir?tme_317
Any downside to using
Plug.Crypto.safe_binary_to_termto filter out any funs? At least that’s what I’ve been doing… and it looks like they have recently extracted plug_crypto to a separate library in case you’re not already using plug itself.Since options are passed to erlang I guess you could use
Plug.Crypto.safe_binary_to_term(bin, [:safe])to protect against atom exhaustion also?michalmuskala
No downside, just that it’s slower than a native option could be since you have to traverse the data after it’s decoded. For now that’s the best option, though.
bettio
Yesterday I wrote a post on this topic, so I would like to share my opinion here https://blog.ispirata.com/how-to-destroy-your-application-using-erlang-binary-to-term-1-575ff7d05333 (I was going to talk about this on the Bert.js topic). Let me know if you don’t agree