A staticly typed API for sending/receiving messages. (In Gleam)

[Crowdhailer]

The idea of a type system checking messages sent and received between processes, has bounced around the erlang(Elixir) community for a while. With Gleam I think we now have an opportunity to explore it.

I have written a module to work with processes in Gleam Below is the section on how to start processes as well as send and receive messages. I’m really interest in feedback particularly in the following areas.

• Is the API ergonomic/clear/easy to use?
• I’m curious how this API’s might be extended to the multi-node case? though I think there is value even without answering that question
• This library works at a lower level that GenServer, (I’m not yet trying to solve those rather build a solid abstraction at this level). Is it useful to work at the spawn/send level or do you think Gleam processes will only be useful once the abstraction level of Gen* has been reached?
• I think Monitors/Link/Supervisors are a second question and one that can be discussed separately in a later forum post

Gleam Processes

You will need these imports for the examples to work

import process/process
import process/process.{Normal, Infinity, From}

Starting and stopping processes

A process is started using process.spawn_link/1.
The single argument is a function that accepts a receive call and returns an ExitReason.

This is an example of the simplest process that immediately exits with reason Normal

let pid = process.spawn_link(fn(_receive) {
    Normal
})

Sending and receiving messages

Here is a simple process that sums all the numbers sent to it.

fn loop(receive, state) {
  let value = receive(Infinity)
  loop(receive, state + value)
}

let pid = process.spawn_link(loop(_, 0))

process.send(pid, 1)
process.send(pid, 2)

By recursively calling loop this function runs forever.
The type of the receive function is parameters by the type of messages it accepts.

Because gleam infers the type of value to be an Int it is known that only integers can be sent to this pid.
The following call will not be allowed by the type checker.

process.send(pid, "0")

Normally a process will accept more message types than an Int,
This is handled by defining a single type with a constructor for each expected message type.

Calling a process

Send a message to a process an await a reply.

type Messages {
  Reverse(From(String), String)
}

fn loop(receive) {
  let Reverse(from, str) = receive(Infinity)
  process.reply(from, string.reverse(str))
  loop(receive)
}

let pid = process.spawn_link(loop(_))

let Ok(reversed) = process.call(pid, Reverse(_, "hello"))

The Reverse message type includes a from reference that says callers accept a String type as a response.
Receiving this message works the same way as before, but now we have a from value that we can send a reply to.
Sending a call message uses process.call/2.
The first argument is a Pid.
The second argument is a function that takes a from reference and creates a message.

Having a constructor for the sent message allows the process library to create a from value.
Doing this allows the calling process to receive replies without having to specify them as part of their message contract.
Again Gleam ensures we can’t send a message that isn’t understood and we can’t reply with a type that isn’t expected.

[lpil]

This is a really interesting design! Thank you for sharing this.

The idea of the From type that sits within the message and is injected by the call function seems really clever! It reminds me of pushing a channel onto a channel so that a response can be had in Go, and it is similar to what I’m currently experimenting with in my channels based design.

I would be interested in seeing how monitoring, linking, and naming would fit alongside this design. In my previous designs I moved away from having receive be a function passed in in exchange for a parameterised Self(accepted_msg) datatype, so type information about the current pid can be shared with the functions that work with links, etc. Receiving a message became process.receive(self, timeout).

Not 100% related to the send-reply design here, but one thing I’m interested in is how we support system messages in Gleam. In the code above they are not supported (in fact they will crash the receiving process) as we’re working at the level of spawn and spawn_link. In my experiments I’ve been making it so the basic process is OTP compatible, so closer to the level of proc_lib.

[josevalim]

@lpil

it is similar to what I’m currently experimenting with in my channels based design.

If you are not familiar with it yet, this paper may be helpful: [1611.06276] Mixing Metaphors: Actors as Channels and Channels as Actors (Extended Version)

[lpil]

Wonderful, thank you

[lpil]

I’ve been thinking about options for monitors and links with this sytem.

It seems like we would want to be able to selectively receive just our accepted messages, just monitor messages, etc. One option might be to expose different functions for these:

// Defined in the process library

pub type ExitMessage {
  Exit(Pid(UnknownMessage), ExitReason)
}

pub type MonitorMessage {
  ProcessDown(Ref, Pid(UnknownMessage), ExitReason)
  PortDown(Ref, Port, ExitReason)
}

pub type SystemMessage {
  System(From, SystemRequest)
}

pub type Receive(msg) {
  Message(msg)
  Exit(ExitMessage)
  Monitor(MonitorMessage)
  System(SystemMessage)
}

// Ideally this type would not be sendable, though the type
// system cannot yet enforce this
pub external type Self(accepted_message)

pub external type Pid(accepted_message)

// In application code

pub type MyMessage {
  MyMessage
}

fn loop(self: Self(MyMessage)) {
  let timeout = timeout.ms(100)

  // Option(msg)
  let msg = process.receive_message(self, timeout)

  // Option(ExitMessage)
  let exit = process.receive_exit(self, timeout)

  // Option(SystemMessage)
  let system = process.receive_system(self, timeout)

  // Option(MonitorMessage)
  let mointor = process.receive_monitor(self, timeout)

  // Option(Receive(MyMessage))
  let msg = process.receive(self, timeout)

  // Option(Dynamic)
  let anything = process.receive_any(self, timeout)
}

With this API we may be able to implement (mostly) OTP compatible processes in a type safe way.

[josevalim]

I would discourage having multiple receive APIs because if someone were to call them one by one, similar to the snippet above, it would unfortunately shuffle the order that messages are received. Or am I missing something?

[lpil]

@josevalim

if someone were to call them one by one, similar to the snippet above, it would unfortunately shuffle the order that messages are received.

That is the intention, this is the equivilent of performing a selective receive.

In application code I would expect these functions to be used about as commonly as a bare receive and the sys functions are used in Elixir. Higher level abstractions similar to GenServer would be normally used, and the functions about would be used to build these abstractions.

[josevalim]

@lpil

That is the intention, this is the equivalent of performing a selective receive.

A selective receive has a unique reference to it, which usually helps enforce that while you can skip ahead, the effect of skipping ahead is quite limited. Selective-receives are also mostly used by “clients”, while system messages are “server” specific.

By providing meal-pieces receives, it means I can accidentally write a code where this:

my_server.cast(Pid, Msg)
sys.debug(Pid)

will invoke the debug code before the cast. It is generally expected that processes will handle messages in the order they are received and this helps break this expectation.

I would personally avoid introducing functionality that makes it easy, even somewhat encouraged, to reorder messages. Some actor languages like Pony don’t even allow you to reorder messages at all - given the implications.

[lpil]

@josevalim

I would personally avoid introducing functionality that makes it easy, even somewhat encouraged, to reorder messages.

I agree. I think all of the above is very much not to be used in application code, and I would consider making it a private module when Gleam has them in future.

We do need some selective receive + link/monitor/system features in order to implement higher level abstractions such as gen_server and supervisor, I’m largely interested in how much of the implementation we can do in Gleam as opposed to Erlang. The API above is unsafe and error prone, but no more so than it would be implementing the same functionality Erlang.

[Crowdhailer]

@josevalim

If you are not familiar with it yet, this paper may be helpful: [1611.06276] Mixing Metaphors: Actors as Channels and Channels as Actors (Extended Version)

Really nice paper. I enjoyed this analogy

“Nonetheless, we will stick with the popular names, even if it is as inappropriate as comparing TV channels with TV actors”

The discussion on the “type pollution problem” of the mailbox is a fair point, although I think it misses (what I see) as the main advantage of a single channel/mailbox is that you can’t get to a state where it is possible to forget to handle messages.
A more subjective point is that I like the fact that a process “owns” what kind of messages it will accept.

A stack implementation is a good example project so I decided to create an example for doing that in Gleam with the process structure I had above.

type Message(a) {
    Push(a)
    Pop(From(a))
}

fn loop(receive, state) {
    case receive(Infinity) {
        Push(new) -> loop(receive, [new, ..state])
        Pop(from) -> {
            let tuple(top, state) = list.pop(state)
            process.reply(from, top)
            loop(receive, state)
        }
    }
}

let s1 = process.spawn_link(loop(_, []))    
let s2 = process.spawn_link(loop(_, []))    

process.send(s1, Push(1))
process.send(s2, Push("hello"))

[Crowdhailer]

Further thoughts:

Receive -> Handle

I consider having function callbacks to handle messages instead of calling receive the main abstraction of the GenServer. There is the possibility of doing the same with an API based on the examples above.

Rewriting the stack example it could look like the following.

type Message(a) {
    Push(a)
    Pop(From(a))
}

pub fn handle(message, state) {
    case message {
        Push(new) -> [new, ..state]
        Pop(from) -> {
            let tuple(top, state) = list.pop(state)
            process.reply(from, top)
            state
        }
    }
}

pub fn init() {
    []
}

gen_server.spawn_link(init, handle)

The function signature of gen_server.spawn_link is spawn_link(fn() -> s, fn(m, s) -> s).

All the messages go through the same handle function, here the Push is equivalent to a cast but Pop is equivalent to a call. I quite like this because the distinction between cast/call is an arbitrary grouping and not related to the business function that each call message might have.

This might make it slightly nicer to handle sys message calls. It is also nice tot to have to pass around a reference to the receive function. All told though I consider the ergonomics of this API is not much improved over the raw process API.

[lpil]

This is very similar to the API I’ve got in my typed message and channels experiments! Perhaps a good omen

[dimitarvp]

Or perhaps you both should join forces.

[lpil]

We have been discussing designs, for sure!

[Crowdhailer]

Links & Monitors

Related to this, but possible a quite separate conversation, I have written down some thoughts on how to handle links and monitors. Better typing for trapped exits and monitor messages · Issue #20 · midas-framework/midas · GitHub

I think the content of that issue could be applied to the receive style or gen server style API.

---

@dimitarvp

Or perhaps you both should join forces.

Perhaps, although deep down I was disappointed to hear that the approaches were similar. My opinion: This area has more fertile grounds for discovery that make it worth branching out.

[dimitarvp]

@Crowdhailer

Perhaps, although deep down I was disappointed to hear that the approaches were similar. My opinion: This area has more fertile grounds for discovery that make it worth branching out.

IMO it has both good and bad sides. Bad, as you said, because nothing much new. Good, because maybe you guys have nailed the mathematically sound (and thus mandatory) parts already and what’s left is mostly stylistic choices. Which bodes well for the future IMO. So keep it up!

[Qqwy]

@Crowdhailer

All told though I consider the ergonomics of this API is not much improved over the raw process API.

Here I tend to disagree. What is the reason you think there is not much improvement?

My two cents are:

• The message type of a GenServer is a sum-type wrapper around a user-provided Message(a) that adds an extra option for e.g. system messages:

type GenericMessage(m) {
  SystemMessage()
  UserMessage(m)
}

• We can capture the type of a GenServer as a product type containing the signatures of the different functions:

type GenServer(opts, m, s) {
  GenServer(init: fn(opts) -> s, handle: fn(m, s) -> s)
}

• And now we can create more higher-level wrapper functions that take a GenServer(opts, m, s) as input, such as the ‘generic handle_info’ that we provide to the Erlang runtime system to handle system messages as expected.
• From other user-fracing code we might just call e.g. gen_server.start_link(my_module.server(), [maybe, some, options])

I think that’s quite powerful .

[Crowdhailer]

All processes termination is “Normal” at the completion of the run function?

@Crowdhailer

This is an example of the simplest process that immediately exits with reason Normal

let pid = process.spawn_link(fn(_receive) {
    Normal
})

I have revised my opinion and think that any return value should be interpreted as exiting normally.
So my example of the simplest process would now be this.

let pid = process.spawn_link(fn(_receive) {
    Nil
})

As the last thing a process is likely do is send a message somewhere, I think it makes sense to have the return value of process.send be Nil drop use of return type from run function when spawning by CrowdHailer · Pull Request #21 · midas-framework/midas · GitHub

@Qqwy

What is the reason you think there is not much improvement?

The function signature you have defined only works for processes that run forever, that might be ok but means tasks need some different construct. Again potentially a plus to separate transient from permanent process. but I like that the run/receive approach works for both.

Also that API forces a separate init step, which isn’t always necessary.

I’m still mulling on how best to handle system messages so I reserve an opinion on that till later.

[Qqwy]

@Crowdhailer

The function signature you have defined only works for processes that run forever, that might be ok but means tasks need some different construct. Again potentially a plus to separate transient from permanent process. but I like that the run/receive approach works for both.

If you want to keep the internals of a handle function pure, then a simple solution would be to extend the return type to allow responses like Noreply(state), Reply(response, state) or Shutdown(), similar to OTP GenServers (but typed).

And on another note, I do think that having separate abstractions for tasks vs. long-running processes makes sense.

[lpil]

@Qqwy

If you want to keep the internals of a handle function pure, then a simple solution would be to extend the return type to allow responses like Noreply(state) , Reply(response, state) or Shutdown() , similar to OTP GenServers (but typed).

How would you type this? I’ve not yet managed to find a way to use this kind of pattern without having all the response values for all the different incoming messages being the same type, which is extremely restrictive in practice.