Functionaly pure message passing

Chat / Discussions
,
1

The Gen* behaviours in Elixir (and erlang) provide a pure interface.

One of the benefits of this is business logic is easy to test. No need to start processes just pass the arguments you want to test to handle_call for example and assert on one of the results.

I’m sure I read in a book that a pure interface was an important part of the setup, unfortunately I can’t remember which one so if anyone can point me to that again I’d be grateful.

Unfortunately with GenServer implemented as it is several things can’t be done in a pure fashion. For example sending a reply to a call in response to a later message.

It’s a fairly contrived example but I don’t believe this can be implemented in a pure manner.

defmodule PairUp do
  use GenServer

  def handle_call({:pair, pid}, from, :none) do
    {:noreply, {:waiting, pid, from}}
  end
  def handle_call({:pair, pid2}, from2, {:waiting, pid1, from1}) do
    GenServer.reply(from1, {:other, pid2})
    {:reply, {:other, pid1}, :none}
  end
end

I was thinking with some small changes to the GenServer design purity could be regained.

defmodule PairUp do
  use AltServer

  def handle_call({:pair, pid}, from, :none) do
    {[], {:waiting, pid, from}}
  end
  def handle_call({:pair, pid2}, from2, {:waiting, pid1, from1}) do
    messages = [
      {from2, {:other, pid1}},
      {from1, {:other, pid2}},
    ]
    {messages, :none}
  end
end

The key changes to this interface is that :send/:nosend are replaced by a list of {target, message} pairings, an empty list giving a same behaviour as no send.

I think the structure {[{target, message], state} could be treated as a writer monad. This might even be a helpful model to add type safety to message sending.

This post is really just me musing. my questions are?

  • Is this a great idea or a horrible idea
  • Does something similar exist already
4 Likes
2

Being idealistic, I think this could be an improvement, but on the other hand, being realistic I know there’s just no chance to change GenServer (and probably for a good reason).

Fortunately, what you propose is already implemented in gen_statem which accepts a list of “actions” in return from state functions:

defmodule Statem do
  @behaviour :gen_statem

  def callback_mode(), do: :state_functions
  
  def init(_args) do
    {:ok, :none, :no_data}
  end

  def none({:call, from1}, {:pair, pid1}, _data) do
    {:next_state, :waiting, {pid1, from1}}
  end

  def waiting({:call, from2}, {:pair, pid2}, {pid1, from1}) do
    {:next_state, :none, :no_data, [{:reply, from1, pid2}, {:reply, from2, pid1}]}
  end
end
2 Likes
3

Completely get this. That said a library implementation could still prove it’s worth.

Thanks for reminding me of gen_statem. It’s existence is a good point.

1 Like
4

Don’t put might-on-will.
Alan Cooper, About Face (1995), p.138

This design rule was specified in connection to user interfaces but over the years I’ve found it to be a good design rule in general. OO often pushed for more generic solutions - frequently in the name of improved reusability - without necessarily accounting for the increase in complexity or decrease in transparency that sometimes results from trying to account for another 5%(? or less) of use cases.

So while your alternate solution may seem more “generic” most use cases only ever need one or no message - not many.

  • Your concern can be easily addressed by composing the logic in handle_call with pure functions which can be tested separately instead of testing handle_call directly.

  • If you are finding that you need to quite often dispatch multiple replies simply go with something like

defmodule PairUp do
  use GenServer
  
  defp multi_reply(msgs), 
    do: 
      Enum.each(msgs, fn({from, msg}) ->
        GenServer.reply(from, msg)
      end)

  def handle_call({:pair, pid}, from, :none) do
    {:noreply, {from, {:other, pid}}
  end
  def handle_call({:pair, pid}, from, msg) do
    multi_reply([{from, {:other, pid}}, msg])
    {:noreply, :none}
  end
end

provide a pure interface.

Can you elaborate on what this means? In connection to functions “purity” is a well defined concept. Going from your post you seem to be largely concerned with message dispatch that isn’t handled via callback return values.

My typical solution is to implement the “business logic” in an entirely separate module and have the GenServer callbacks simply invoke those module functions. So PairUp would be simply GenServer interaction logic (and GenServer related helper functions) while some PairUpLogic module would contain the actual “pure business functions”. So testing would focus on PairUpLogic while PairUp callbacks would mostly just call PairUpLogic functions.

5 Likes
5

PairUpLogic This seams like an abstraction above what I want. In my opinion the module using GenServer is already the business logic module. With Genserver hiding details about sending monitoring etc.

That is my main concern I want to have all message dispatch handled by callback return values

6

I would agree that in most cases, that is sufficient. If there were a very complex business process, however, which may require a lot of complex rules about sending multiple replies or deferring some replies, it could be beneficial to write a separate pure module with business logic, which would provide in a pure, declarative form “instructions” for the GenServer to carry out, facilitating better testing of that complex logic. However, most GenServer implementations will not need that complexity.

1 Like
7

So I have written an implementation that allows an arbitrary number or messages to be sent in response to a received message. Those messages can also be sent to new processes and not just part of a GenServer reply.

https://hexdocs.pm/comms/0.1.0/Comms.Actor.html

The README has the long term plan explaining why try so hard to make all my message passing explicit.

The ultimate goal is a program that has deep insight into the message patterns in a program. This insight should be sufficient to warn of race conditions or deadlocks. By using Elixirscript these insights should extend to client server interactions.

8

I’m unconvinced by what you’re aiming to do with this library. You say in the docs that

This allows an Comms.Agent to send any number of replies rather than 1 or 0 that is the case for GenServers.

which isn’t really accurate—you can’t ever send more than one reply, since a reply implies that someone is waiting for it—a GenServer.call will block until a reply is received. If another message is sent apart from the reply, it’s either just another asynchronous message to be handled in some way, or if it’s also structured like a GenServer reply, it will sit in a process mailbox forever since call attaches a unique reference to its message so that it can recognise the reply.

This is also the reason that GenServer explicitly distinguishes between calling and casting—a call will block until a reply is received, and a cast cannot be replied to.

So, when you say that your module makes it possible to send multiple ‘replies’, it doesn’t. It likely allows you to send multiple outbound messages based on one incoming one, but so does a normal GenServer. ‘Unifying’ all the handler functions into one will likely mean that any potential users would likely recreate things like synchronous casting and asynchronous calling themselves, likely in a much less robust way than the built-in GenServer functions.

Perhaps I’m misunderstanding your project, and if so, correct me, but I just don’t see what you’re trying to achieve. You do mention that you’d like to

make all […] message passing explicit

and

warn of race conditions or deadlocks

but I don’t see how this project helps you towards this. Firstly, you can actually inspect all messages flowing through a system using built-in BEAM functions—just because the user does not explicitly send messages, does not mean they are not ‘explicit’ if your goal is system analysis. Secondly, surely you don’t expect the entire system to only ever run these Comms.Actors? Even if you were to convince a user to only use them to build out functionality, many libraries will be using regular BEAM functionality to achieve their goals, and so even if you did manage to have some run-time model and insight into all the actors running, the messages between them and any potential deadlock or race-condition scenarios within them, you could not say the same for interactions between the user system and the libraries it will inevitably use.

9

In my original PairUp example I send zero replies in response to the first call and two in response to the second call.

I should have been more explicit. I want to reason about messages at compile time.

10

Ok yeah, I see what you’re doing. You’re moving the async calls to sending messages and replying to pending calls into an actual function return as opposed to doing it ad-hoc.

The problem still remains of how your code interacts with other code if you want to globally reason about messaging. I think only a small amount of user code ever uses more complex or advanced configuration of GenServers to the point where trying to reason about deadlock or race conditions wholly within your own system would be fruitful, but best of luck with your project.

To clarify as well, when I said that if I had to use multiple asynchronous replies etc. in my codebase, splitting up the code into pure business logic and a GenServer to interpret, I would have my business logic reply in some business-logic specific way and let the GenServer translate that into whatever complexity is needed, whereas here you seem to be pushing towards enforcing a GenServer-like-yet-not-standard convention on the user’s business logic itself. If you do manage to get some static reasoning about messages together I would be very interested in looking at that code, but I still remain unconvinced of how useful it would be for actual development.

1 Like
11

Just as an aside, if you think of the BEAM Actors and Message Passing as just a normal Algebraic Effect (which it is), then it is functionally pure. :slight_smile:

12

So it is!

And it turns out someone has done some work on formally modelling and verifying BEAM message passing already.

2 Likes
13

Hah nice! That’s new! :slight_smile:

14

Yep that is the kind of thing im looking for

15

This quote is pretty close to saying GenServer is valuable because of the ability to test by just calling one of the handle_calls and checking the return values. (Not true in every case, e.g. GenServer.reply/2)

Another interesting thing about what we did when separating the generic from the specific is that we instantly made it much easier to test our individual modules. If you wanted to unit test the old kitty server implementation, you’d need to spawn one process per test, give it the right state, send your messages and hope for the reply you expected. On the other hand, our second kitty server only requires us to run the function calls over the ‘handle_call/3’ and ‘handle_cast/2’ functions and see what they output as a new state. No need to set up servers, manipulate the state. Just pass it in as a function parameter. Note that this also means the generic aspect of the server is much easier to test given you can just implement very simple functions that do nothing else than let you focus on the behaviour you want to observe, without the rest.

16

Couldn’t you use an arbitrary {to, tag}?

i.e.

def test_gen_server_call(msg) do
  ref = make_ref()
  Server.handle_call(msg, {self(), ref}, nil)
  receive do
    {^ref, reply} -> assert_this_reply_is_correct(reply)
  end
end
1 Like
17

I stumbled across this a few times while writing Elixir code. I separated the code into processes that return actions, and processes that execute actions… I did not go further because it was a application that integrated GitHub and Slack and what really broke purity was HTTP calls. And writing a framework for doing it went well beyond the scope of the project.

That said I think Elm has a great approach to managing effects purely and could be a good inspiration :slight_smile: I would love to collaborate on something similar for the backend

Now from a theoretical perspective, recursive types offer a basis for purely functional processes, that might be a solid starting point for more academic work, so I thought it is worth mentioning.

Roughly, a process is a function that takes a message and returns a reply and a new process (which is again a function that takes a message and … all the way down):

process :: message -> (message, process)

Does anyone know if dyalizer can check this kind of types?

18

So I have though about this quite a lot.

  • It would be necessary to be able to send more than one message in response to a single incoming message
  • You need to specify the address and the message
  • An address needs to me more generic than a pid, i.e. it could be a http endpoint.

Finally having http as pure instructions where what led me to develop raxx. To model it like this you need a data structure for the request/response but that is a little of the core topic.

I think all the above points can be address with the following signature

spec handle(message, state) :: {[{address, message], state}

for example.

def handle({%HTTP.Request{path: "sign_up", body: email_address, from: client}}, state) do
  email = %Email{body: "Welcome name", target: email_address}
  mailer = state.mailer

  response = %HTTP.Response{status: 201}
  
  {[{mailer, email}, {client, response}], state}
end

Re dialyzer.

Because more than one spec can be added to a function I think you could specify a lot of information about the protocol

spec handle(:m1, :s1) :: {[:a, :m2], :s2}
spec handle(:m2, :s2) :: {[], :s1}

This says that the handle function does NOT handle the message :m1 when in state :s2

Addresses

I think addresses could be the same as you can use for genservers i.e. pid | atom | {:via, module, term}

comms

I have run some very basic experiments in this project. feel free to ignore, i didn’t get very far but would like to revisit the project one day GitHub - CrowdHailer/comms: Explicit message passing for improved reasoning about actor systems

1 Like
19

Oh i’ll take a look at it. Did you ever try Elm? It is pretty nice to play with

20

It may be the case that the reply never comes if, e.g. the GenServer is (logically) waiting for some other call or cast before it decides to reply to the original caller.