What is erl_interface?

I know that broadly, the two ways Erlang interacts with other langues are NIF / Ports. My question is: how does erl_interface http://erlang.org/documentation/doc-5.2/pdf/erl_interface-3.3.2.pdf fir into all this ?

If I understand this correctly, programs implementing Erl_Interface can talk to epmd, and, as a result, behave like a normal distributed Erlang node.

Question: is erl_interface a third alternative to NIF / ports, or is it done via NIF / ports ?

Thanks!

A relatively shallow answer: this is basically a protocol that allows non-Erlang nodes to participate in Erlang distributed mesh. I heard of implementations in Go and JS.

1 Like

This chapter of the docs talks about the different ways to interface with Erlang.

Erl_Interface
The program at the other side of a port is often a C program. To help the C programmer, the Erl_Interface library has been developed

The Erlang external term format is a representation of an Erlang term as a sequence of bytes, that is, a binary. Conversion between the two representations is done using the following BIFs:

Binary = term_to_binary(Term)
Term = binary_to_term(Binary)

A port can be set to use binaries instead of lists of bytes. It is then not necessary to invent any encoding/decoding scheme. Erl_Interface functions are used for unpacking the binary and convert it into a struct similar to an Erlang term. Such a struct can be manipulated in different ways, be converted to the Erlang external format, and sent to Erlang.

1 Like

Here are a number of things I don’t understand:

  1. port operates over stdio right ?

  2. in this case, port can also be binary? (I’ve never seen stdin / stdout be binary, but I guess there is no reason it can’t be binary)

  3. erl_interface allows programs to be nodes in distributed erlang right ? how is that possible if communication is over stdio instead of tcp ?

You can use file descriptors 3 & 4 instead of stdio over a port. See open_port docs for more info. If you use binary for a port Port.open({:spawn_executable, exe}, [:binary, packet: 4, ...], you encode with binary_to_term when you send to the port, and decode with term_to_binary when you receive. In the port, you have to encode/decode too.

erl_interface is a bunch of c functions for encoding/decoding external term format, communicating with nodes etc. Boilerplate if you’re writing something in C that you want to interface with Erlang.

You can implement the network protocol w/o using erl_interface, ala Pyrlang.

@cmo: I just got GitHub - tomaon/ei: erl_interface for rust working .

Could you take a look at that and let me know if this is basically the “erl_interface” we are talking about above ?

Basically, is erl_interface “just” binary-erlang-terms over port ? Is there more to this ?

You don’t need to use erl_interface to communicate to erlang/elixir through a port from the c side, if you use another serialization format that are understood by both, such as JSON.

With erl_interface, you can use the native erlang term format for port communication in the simplest case. In a much more complex case, you can even pretend to be a erlang node (c-node) like @dimitarvp mentioned.

Are you repeating documentation or speaking from actual experience with erl_interface ?

I ran the examples at GitHub - tomaon/ei: erl_interface for rust and am wondering if this is “all there is” to erl_interface, or if there is something more.

erl_interface can be used for many different purposes. For example:

  • Write a program in C/C++ that behaves like a distributed Erlang node and communicates with other Erlang nodes.

  • Simplify the implementation of a port program and the Erlang code that communicates with the port program. The Erlang code can use just term_to_binary/1 when sending data and binary_to_term/1 when receiving data (as opposed to having the Erlang code use the binary syntax to decode and encode binary data). The same thing can be used in a driver.

  • Write a C/C++ program that reads a Wings 3D file, decodes the Erlang term inside it, and displays the 3D model or converts it to some other 3D file format. In the opposite direction, a C/C++ program could convert another 3D file format to a Wings 3D file.

Where I said Erlang, you can substitute Elixir or any other BEAM language.

3 Likes

@bjorng : Here is my XY problem. I have some code in Rust. I have some code in GoLang. I want to
use Elixir to provide a REPL and supervise/orchestrate/manage everything.

One obvious solution is:

  • use Protobuf + generate bindings for Go/Rust
  • use TCP for everything
  • for each GoLang/Rust kernel-process, have a corresponding Elixir ‘translator’ process who’s sole job in life is to translate Elixir data <-> Protobuf, and send it over TCP.

I am looking to see if there are better Elixir/Erlang based solutions. So far, I see: Nif, Port, erl_interface. erl_interface, as far as I can tell right now, looks like binary erlang terms over stdio.

Question: do you have any advice regarding this? I am increasingly leaning towards the protobuf/TCP solution, but I feel I don’t understand the existing Erlang/Elixir solutions well enough to discard them.

Thanks!

I am not sure what you mean by “use TCP for everything”. Do you mean Erlang distribution over TCP, or plain binary data over TCP with your own encoding/decoding code, or something else?

I don’t know enough about Go and Rust, or how complicated data you will need to send, but here are some thoughts:

  • If it is possible to link the erl_interface library to Go and Rust executables (or if the Rust version of erl_interface supports Erlang distribution), you could have those executables pretend they are Erlang nodes. That will make the code on the Erlang/Elixir side very simple. It may make the Go/Rust code more complicated.

  • If you use a NIF (assuming that you can link Go code into a NIF; I know you do it with Rust), the Erlang system could crash or become unstable if there are any bugs in the Go/Rust code or any of your interface code). If that is problem, you should not use a NIF.

  • Whether you use port programs or a plain TCP socket doesn’t really matter. Either way, you will need some way to encode/decode your data. You could use erl_interface on the Go/Rust side, and term_to_binary/binary_to_term on the Erlang side, or you could use any other way to encode/decode the binary data.

To some extent, which solution you choose depends on where you want to complicate the interface code. Using erl_interface will make the Erlang/Elixir code simple, but will probably make the Go/Rust code more complicated.

That will probably work well. I don’t know much about protobufs, except that there is an excellent Erlang library for protobufs (which I assume you already know).

3 Likes

Thank you for the insightful response.

problem statement

As a review of what’s going on, I have some server components in GoLang. I have some server components in RustLang. I want to glue / manage everything from Elixir. The core question is: how does Elixir-Go communicate and how does Elixir-Rust communicate. For simplicity, let’s only consider the Elixir-Rust case.

known solutions

My known options are (a) NIF, (b) Port, (c) erl_interface, (d) protobuf over tcp.

For 2a / NIF, we would use something like GitHub - rusterlium/rustler: Safe Rust bridge for creating Erlang NIF functions . As mentioned above, this has the danger of Rust crash → Beam crash, so let us rule out this option.

I don’t think I have tried 2b/Port yet.

For 2c/erl_interface, one solution is GitHub - tomaon/ei: erl_interface for rust . I tried it. I am not convinced of the benefits yet.

For 2d/protobuf_tcp: Elixir and Rust would run as separate processes, talk over protobuf over tcp. I intend to use GitHub - elixir-protobuf/protobuf: A pure Elixir implementation of Google Protobuf. for elixir protobuf. I don’t think this matters, but mentioning it in case it does.

2d is also what I meant by when I said “use TCP for everything”

questions

question 1: is erl_interface merely binary-erlang-terms over stdio ports ?

when I run the example at ei/README.md at main · tomaon/ei · GitHub , that is all I see – stdio ports + binary serialization of erlang terms

question 2: does erl_interface have some epmd / run as independent node functionality ?

I have setup distributed Erlang with multiple nodes and multiple machines and played with epmd. erl_interface claims to make GoLang/RustLang code an “erlang node” – but in the example above, all I see is binary-erlang-terms over stdio ports – i.e. the Go/Rust code is still “wrapped” by an Erlang process.

Does erl_interface allow Go / Rust to run as “independent Erlang node”, without being first wrapped by an Erlang process ?

question 3: what is the advantage, if any, of erl_interface over protobuf-over-tcp ?

This is a really insightful point, and I am currently leaning heavily in favor of pushing all the complexity to Elixir, as Elixir is serving as the ‘glue’.

1 Like

I think most of that is answered above and in the links.

You can use ports without erl_interface, but you’ll want a library in that language that will encode/decode the external term format. Go has at least one. Sasa’s post on ports is a nice primer.

This is dismissive and patronizing. What is the answer to q2? Can erl_interface allow Go/Rust programs to run as distributed Erlang nodes without first being wrapped by an Erlang process?

Both. Your choices are wide open. I will start from port and skip erl interface, just use json. This should be even easier than say protobuf. Once you hit some perf bottleneck you can always do more.

Using erl_interface, were you ever able to write a non-Erlang/Elixir program that functioned as an distributed Erlang node, without first being wrapped by an Erlang process ?

Your latency / throughput requirements != my latency / throughput requirements. As stated, most of the servers are already in GoLang / Rust.

Haven’t try that. I guess it is pretty involved. I use ports. It is a library that contains the necessary functions; you link it with your program so there will be no Erlang process involved.

Of course. So you need to judge for yourself. The possibility is there.

Can you point me at demo code that does this: use erl_interface to have non-Erlang program act as a distributed Erlang node, without having it be wrapped by an Erlang process.

It is in erl_interface user’s guide

2 Likes