Functional Web Development with Elixir, OTP - Is OTP part of the entity layer?

Now while it’s nice to not have the logic conflated with the GenServer ceremony it can get a bit boilerplate-y when you are dealing with numerous GenServers.

One compromise:

  • be more organized inside the GenServer callback module so that it is very clear what is what
  • in the test suite define some helper functions that make it easier to test the callback functions.

That way the “conflation” can be a bit less distracting.

# alias IslandsEngine.{Demo, Rules}
# state = Demo.init("Miles")
# {:reply, :error, state} = Demo.guess_coordinate(state, :player1, 1, 1)
# {:reply, :ok, state} = Demo.add_player(state, "Trane")
# {:reply, :ok, state} = Demo.position_island(state, :player1, :dot, 1, 1)
# {:reply, :ok, state} = Demo.position_island(state, :player2, :square, 1, 1)
# state = %{state | rules: %Rules{state: :player1_turn}}
# {:reply, {:miss, :none, :no_win}, state} = Demo.guess_coordinate(state, :player1, 5, 5)
# {:reply, :error, state} = Demo.guess_coordinate(state, :player1, 3, 1)
# {:reply, {:hit, :dot, :win}, state} = Demo.guess_coordinate(state, :player2, 1, 1)

defmodule IslandsEngine.Demo do
  alias IslandsEngine.Game

  ### helper functions for testing - i.e. should be under "test"" ###

  def init(name) do
    {:ok, state} = Game.init(name)
    state
  end

  def add_player(state, name),
    do: Game.handle_call({:add_player, name}, self(), state)

  def position_island(state, player, key, row, col),
    do: Game.handle_call({:position_island, player, key, row, col}, self(), state)

  def set_islands(state, player),
    do: Game.handle_call({:set_islands, player}, self(), state)

  def guess_coordinate(state, player, row, col),
    do: Game.handle_call({:guess_coordinate, player, row, col}, self(), state)
end

defmodule IslandsEngine.Game do
  use GenServer

  alias IslandsEngine.{Board, Coordinate, Guesses, Island, Rules}

  # --- GenServer Client API ---

  @players [:player1, :player2]

  def add_player(game, name) when is_binary(name),
    do: GenServer.call(game, {:add_player, name})

  def position_island(game, player, key, row, col) when player in @players,
    do: GenServer.call(game, {:position_island, player, key, row, col})

  def set_islands(game, player) when player in @players,
    do: GenServer.call(game, {:set_islands, player})

  def guess_coordinate(game, player, row, col) when player in @players,
    do: GenServer.call(game, {:guess_coordinate, player, row, col})

  # --- GenServer Ceremony ---

  def via_tuple(name),
    do: {:via, Registry, {Registry.Game, name}}

  def start_link(name) when is_binary(name),
    do: GenServer.start_link(__MODULE__, name, name: via_tuple(name))

  def init(name),
    do: {:ok, game_init(name)}

  def handle_call({:add_player, name}, _from, state),
    do: handle_add_player(state, name)

  def handle_call({:position_island, player, key, row, col}, _from, state),
    do: handle_position_island(state, player, key, row, col)

  def handle_call({:set_islands, player}, _from, state),
    do: handle_set_islands(state, player)

  def handle_call({:guess_coordinate, player, row, col}, _from, state),
    do: handle_guess_coordinate(state, player, row, col)

  def handle_info(:first, state) do
    IO.puts("This message has been handled by handle_info/2, matching on :first.")
    {:noreply, state}
  end

  defp reply_success(state_data, reply), do: {:reply, reply, state_data}

  # --- Game Module Logic

  defp game_init(name) do
    player1 = %{name: name, board: Board.new(), guesses: Guesses.new()}
    player2 = %{name: nil, board: Board.new(), guesses: Guesses.new()}
    %{player1: player1, player2: player2, rules: %Rules{}}
  end

  defp handle_add_player(state, name) do
    with {:ok, rules} <- Rules.check(state.rules, :add_player) do
      state
      |> update_player2_name(name)
      |> update_rules(rules)
      |> reply_success(:ok)
    else
      :error -> {:reply, :error, state}
    end
  end

  defp handle_position_island(state, player, key, row, col) do
    board = player_board(state, player)

    with {:ok, rules} <-
           Rules.check(state.rules, {:position_islands, player}),
         {:ok, coordinate} <-
           Coordinate.new(row, col),
         {:ok, island} <-
           Island.new(key, coordinate),
         %{} = board <-
           Board.position_island(board, key, island) do
      state
      |> update_board(player, board)
      |> update_rules(rules)
      |> reply_success(:ok)
    else
      :error ->
        {:reply, :error, state}

      {:error, :invalid_coordinate} ->
        {:reply, {:error, :invalid_coordinate}, state}

      {:error, :invalid_island_type} ->
        {:reply, {:error, :invalid_island_type}, state}
    end
  end

  defp handle_set_islands(state, player) do
    board = player_board(state, player)

    with {:ok, rules} <- Rules.check(state.rules, {:set_islands, player}),
         true <- Board.all_islands_positioned?(board) do
      state
      |> update_rules(rules)
      |> reply_success({:ok, board})
    else
      :error -> {:reply, :error, state}
      false -> {:reply, {:error, :not_all_islands_positioned}, state}
    end
  end

  defp handle_guess_coordinate(state, player, row, col) do
    opponent_key = opponent(player)
    opponent_board = player_board(state, opponent_key)

    with {:ok, rules} <-
           Rules.check(state.rules, {:guess_coordinate, player}),
         {:ok, coordinate} <-
           Coordinate.new(row, col),
         {hit_or_miss, forested_island, win_status, opponent_board} <-
           Board.guess(opponent_board, coordinate),
         {:ok, rules} <-
           Rules.check(rules, {:win_check, win_status}) do
      state
      |> update_board(opponent_key, opponent_board)
      |> update_guesses(player, hit_or_miss, coordinate)
      |> update_rules(rules)
      |> reply_success({hit_or_miss, forested_island, win_status})
    else
      :error ->
        {:reply, :error, state}

      {:error, :invalid_coordinate} ->
        {:reply, {:error, :invalid_coordinate}, state}
    end
  end

  defp player_board(state_data, player), do: Map.get(state_data, player).board

  defp opponent(:player1), do: :player2
  defp opponent(:player2), do: :player1

  defp update_player2_name(state_data, name), do: put_in(state_data.player2.name, name)

  defp update_board(state_data, player, board),
    do: Map.update!(state_data, player, fn player -> %{player | board: board} end)

  defp update_rules(state_data, rules), do: %{state_data | rules: rules}

  defp update_guesses(state_data, player_key, hit_or_miss, coordinate) do
    update_in(state_data[player_key].guesses, fn guesses ->
      Guesses.add(guesses, hit_or_miss, coordinate)
    end)
  end
end
2 Likes

That is precisely the question :smiley:

I understand this point. However I still have the following question:

  • Given that the core should be functional, should we care about processes at all? Isn’t a process a runtime requirement that is independent from the functional requirements (the core of the application)?

I ask this, because when I used to work with Java (yes yes, I can hear you mocking me, that’s fine :P) we used to focus our application in the functional requirements, and only then if needed, we added threads for concurrency. I understand Java threads are a poor comparison to erlang processes but the idea I am trying to take out of here is how to organize workflow and to separate runtime concerns from other concerns.

I am not saying I don’t agree that we need a process. I think this is a sound solution that makes a whole lot of sense (and now that Supervisor follow, it makes even more sense) I am just asking if when you build application cores and, you already think about processes.

Perhaps this image helps explain my doubts:

So, I have the idea that Island, Guesses, Board, Coordinate, Rules and Game all belong to the core layer. OTP would be the layer on top, the Domain one, where we add processes, supervisors and support for fault tolerance / concurrency.

Perhaps I am miss-interpreting the lessons in the book, perhaps this onion model doesn’t really fit (perhaps both), but I am struggling to understand where each piece fits. This struggle is where my questions come from :smiley:

@peerreynders He he, yeah, I would personally separate them. But now I am keen on knowing the disadvantages of coupling to OTP behvaiours (topic for another discussion).

Thanks for the code though, that is exactly the module I was referring to !

Usually to satisfy some non-functional requirements. And when threads are introduced the structuring of the logic is impacted, sometimes severely.

In the end OTP is simply a standard library containing “formalizations of common patterns” of using the underlying virtual machine - it’s not some external system.

The other difference is that with threads you focus on flow of control while with processes you focus on the flow of messages (data). That will fundamentally impact how you structure your logic.

With the BEAM processes exist inside the core and domain of the onion diagram, just like in Java the threads package wouldn’t be pushed into the API layer (unlike lets say java.sql which is only used to communicate with an external system, a database).

Given that the core should be functional

Where is this coming from?

With the BEAM there is sequential programming and concurrent programming. Sequential programming is functional programming and happens strictly inside a process. Concurrent programming is about communicating processes.

The whole point of the BEAM is that you use both styles in concert.


Let’s turn this around. What in your view disqualifies a process from being an entity?

2 Likes

The fact that it mixes runtime concerns with functional concerns. So, how would you classify the structure of the Islands project in the book? Where would you put Island, Rules (state machine) Guesses, Board and the Game entity?

All together in the core? All together in the Domain? Separated (if so how?)
Perhaps you would use another Onion Diagram instead of the one I proposed?

Porting to Erlang is easy as Elixir more or less runs on top of Erlang and all the system building concepts like OTP come from Erlang.

When a method is used to access or mutate a class instance it does so under the flow of control of a thread implicitly. It’s only after you decide that you need distinct flows of control i.e. multiple threads that they become explicit.

Process oriented design addresses these type of runtime concerns deliberately at design-time by always making them explicit.

Where would you put Island, Rules (state machine) Guesses, Board and the Game entity?

The domain. But you have to accept the fact that “entities” can manifest in different forms. In Java it may be a single class instance or an aggregation of instances. On the BEAM an entity could simply be represented as a data structure, a process, or multiple collaborating processes. As the earlier code has shown, you can view a process as an aggregate of its state and the process shell.

All together in the core? All together in the Domain?

The relationship between the Domain and the Core is muddy at best.

  • here the application core cuts across domain model, domain services, and application service.
  • here Application + Domain are lumped together.

So the nature of the boundary between Core and Domain isn’t exactly clear. The Core at times comes across as some kind of standard library of non-domain specific building blocks. So are we talking about arrays, lists, maps, etc? :man_shrugging:

Then on the BEAM processes are in the core, as is OTP.

3 Likes

My first impression here is that I’m wary of trying to apply ideas from other ecosystems wholesale onto the BEAM ecosystem. When I was new to Elixir, I did it, and I know it did not help me.

Part of the idea of the book is to explore what’s possible in this new world. There are things that we can do on the BEAM that you would be hard pressed to do in other languages/ecosystems. Trying to apply rules/approaches/schemes from those other ecosystems onto this one seems unhelpful.

My strong preference would be to look at what we’re doing in the book and judge it on it’s own merits rather than comparing it to other systems or trying to fit it into a paradigm from another ecosystem.

That’s me trying to say, let’s please leave Java at the door. :^)

I’m also reminded of that old joke about the best answers to software questions always beginning with, “It depends.”

To me, questions like the one we’re looking at come into the realm of design decisions based on the constraints of the individual project. As we have no doubt seen, the only constant in software work is change.

The most important thing, then, is to design things with flexibility in mind. The individual choices we make are important, but the ability to easily change what we’ve done seems to me to be even more important.

Early in the book, we talk about having the ability to run thousands and thousands of games on the same node. I take that a constraint on the system. We need to solve for that in the design.

You’ve agreed that when we build a GenServer, we’re really defining a module and functions that work on some data. The change is that we decided that those functions should all run in a separate process (or processes).

To me, that says that we’re still working with the same things we were working with when we defined Board, Island, Guesses, and Coordinate, but by making this piece a GenServer, we satisfy the need for many games on the same node.

By using GenServer, we just did the two things at once - define the entity with a module, some functions, and some data as well as allow us to run it in multiple processes.

Here’s where your personal choice comes in. If this makes you feel uncomfortable, you can absolutely use the code for an intermediary module that @peerreynders wrote up. All will be well.

To me, the key point is that the design we have in the book makes that refactor a straight forward affair. You could do it wholesale. You could do it one api function/callback at a time.

It’s your choice. You can do it in response to changing requirements. You can manage the transition safely.

8 Likes

After reading everyone’s replies, I must say I am confused.

I understand we ought to leave Java at the door - I find that perfectly reasonable and I agree (after all, Elixir/Erlang has a different kind of OO than Java because they were more influenced by SmallTalk and their first version/prototype was done in Prolog iirc).

But at the same time, I find it very hard to understand, that a decade of architectural design patterns should just be thrown out of the window because Elixir is new. I mean, surely there are some concepts, some links we can make from the most common layered architectural design patterns right? Sure, maybe the onion one is a terrible pattern to apply in Elixir, perhaps the hexagonal one is way better because it only has 3 zones (application, DMZ where the adapters are and the OHW, aka, outside horrible world) but we should still have some sort of resemblance going on.

I say this, mainly because I am with Pragmatic Dave when he says “Building applications in INSERT_TECHNOLOGY_X_HERE doesn’t change how good applications are built because the basic good core tenets of architecture are the same no matter which technology you use”.

I am not saying the book example is lacking a good architecture. I am just saying I am personally having trouble comparing the tenets of well known architectural patterns to the example and connecting the dots.

This is not a problem with the book, this is a problem with me, I am very well aware of that - I am quite obsessed with design and architecture and the main reason I want to get it right in Elixir is because I have been through years and years of suffering from bad design choices made by others and I have grown honestly rather tired. I am just trying to keep that cycle from repeating itself and I know that starts with me, but first I need to understand where all the pieces fit together.

In your opinions, is there any architectural pattern that somehow resembles what the example in the book follows? Is this is a new approach that is seen in most Elixir applications and that I missed because I am still learning Elixir?

1 Like

I guess I’m confused what you mean when you say the “entity” layer. In the context of the book, the games are built as data structure. They take in new data and provide a new data structure back to you. In order to take these data structures and make them into a useful game we put the data structure in a process and allow the process to manage the lifecycle of that game. I’m really not sure where the hangup is I guess.

As to throwing things out, I don’t think anyone is throwing out any established architectural patterns here. The book provides an alternative to all of the patterns you mentioned. At least I think it does. Out of all the patterns you’ve mentioned the only I’d ever heard of was “hexagonal” (a pattern that I’m not a fan of tbh). I don’t know what “onion” or “lasagna” architectures are but I guess they involve layers?

Either way I don’t think that you should take the book as the end all be all to design. If you want to just put stuff in a database that’s quite alright and that concept will generally scale well and lead to less stateful designs. But if you need to bring state into your system then the book shows you a way to layer together a functional interface with stateful processes.

5 Likes

Is there a “shortage” of Erlang/Elixir patterns? Maybe, maybe not:

http://erlang.org/pipermail/erlang-questions/2007-October/030091.html

Garrett Smith attempted an Erlang specific repository but contributions stalled.

And even if a prescriptive pattern was followed - it could still be wrong if it is applied in the wrong context (this is a real problem in OOP with respect to design patterns).

Ultimately Game is just a process manipulating some data structures (Board, Coordinate, Guesses, Island, Rules) - and Game itself is going to be part of a supervision tree which itself is part of an OTP application.

For a discussion on how to structure supervision trees: The Hitchhiker’s Guide to the Unexpected
Data structure vs Process: To spawn, or not to spawn?
Structured Programming Using Processes (2004)

Building applications in INSERT_TECHNOLOGY_X_HERE doesn’t change how good applications are built

Yes, but going from Java to C# isn’t changing the paradigm so the solution shape isn’t impacted. But going to Erlang/Elixir you are shifting paradigms into “Concurrency Oriented Programming” which means that processes and most of OTP becomes part of the Core rather than being a

What seems to currently be challenging you the most is that a process can be an entity.

I’ve commented on the onion architecture before and in my view the most insightful aspect was that the domain layer gets to dictate the contracts while the infrastructure has to implement those contracts (e.g. nothing in the data store is influencing what is going on in the domain).

That particular advice still applies to Elixir - if that is the type of system that is being built.

Most of these architectures boil down to examining the structure and characteristics of a system along certain dimensions:

  • Boundaries
    • at all levels of granularity (types, user defined types (modules), processes, supervision trees, applications, systems)
    • physical vs logical boundaries
    • (high) cohesion of logic within any one boundary
    • (low) coupling between boundaries
      • dependencies from within the boundary
      • dependencies on the boundary
      • i.e. good inter-boundary contracts (hiding implementation details, correct (specification → implementation) direction, etc.)

Alongs those lines what are the problems that you see with the design of Game?

Now there are some games you can play to hide whether or not something is a data structure or a process. But processes that are part of a supervision tree are known to be processes - otherwise they couldn’t be part of the tree.

6 Likes

First, I want to echo what @keathley and @peerreynders had to say. They covered a lot of good ground pretty thoroughly, so I’m going to take another direction in trying to answer your question.

I’d like to start by making a distinction between design patterns and design principles.

The Gang of Four book is probably the most famous source of design patterns, but the architectural patterns you mentioned - hexagonal and onion - are as well. My sense of design patterns is that they’re fairly strict. There are rules to follow. There are definitions and technical terms to learn.

Design principles are more loose and general. These are often just phrases like “build in layers”, “minimize complexity”, “name things expressively”, and so on.

Design principles most often use common terms with their everyday meanings. They tend not to emphasize hard and fast rules, but this doesn’t make them any less useful. In fact, their flexibility makes them applicable in a broader range of cases.

That flexibility gives the developer a lot of discretion, but it requires the developer to exercise more of their own judgement as well.

Here’s where this applies to the concern you raised that we might be throwing hard won knowledge out the window.

When folks started to write Gang of Four style books for other languages (GoF was originally written for C++ iirc), they needed to change and adapt the patterns. Sometimes they needed to make considerable changes. Sometimes they needed to throw patterns away completely because they didn’t make any sense in the new target language. (I’m thinking of the Ruby one especially.)

This reinforces the idea that patterns aren’t very flexible. It’s really hard to apply them across languages. It’s way harder to apply them across the OOP/FP border.

Principles are much more likely to make that trip successfully. My suspicion is that in the line you quoted from Dave Thomas, he’s talking about something closer to principles than patterns. (But that’s only my suspicion, and I don’t want to put words in Dave’s mouth.)

Also in this respect, Elixir isn’t really new, as you say. Elixir semantics are very close to Erlang’s, and Erlang has been around for decades. Elixir and Erlang are just different from most other paradigms. So this idea that we can readily apply patterns, not principles, from OOP seems really unlikely to me.

4 Likes

Ahhh, I think I get it now.

All this time I have been investing my time in reading Patterns and Rules, not realizing that Erlang’s landscape requires a different approach from the get go.

After reading evryone’s replies, I have concluded the following:

  • I need to focus more on architectural design principles and not patterns (specially the ones translated from other languages)
  • I need to invest more on books and courses on how to model applications in Elixir (I suppose it doesn’t change much from erlang, right?)

When I started the book I had a copy of the GoF design patterns right alongside it. I still do, it’s a book I cherish a lot. But I now understand that the Elixir best characteristics are better fostered in a environment not restricted by the patterns of 15 years ago, made for C++.

In a way, this is both humbling and freeing. Humbling because I have cemented my position as a beginner. It somehow feels like after years of being a Senior in other areas where I had nothing more to really learn, coming to Elixir is almost like going back to level 1.

But I also feel free, because now that I understand this, I am more open minded and reading new books and posts from the community will be easier for me. I am not perfect, there will always be a part of me that goes “This really looks like pattern X or goes against rule Y” but now I feel I don’t have to compare everything against this huge database of past experiences and accepting new information (while still applying critical thinking ofc) will be easier.

It also makes me feel better about buying more books about Elixir, so I guess it’s good for everyone :stuck_out_tongue:

When folks started to write Gang of Four style books for other languages (GoF was originally written for C++ iirc), they needed to change and adapt the patterns. Sometimes they needed to make considerable changes. Sometimes they needed to throw patterns away completely because they didn’t make any sense in the new target language. (I’m thinking of the Ruby one especially.)

This is an important piece of history I didn’t know of and I can only wish I had heard it before!

2 Likes

Consider it the more senior position to be able to answer with “It depends” and being able to select the way forward given your own constraints and not sticking to patterns just for the sake of it.

1 Like

Just a couple of comments. I also like patterns, but I understood they were guidelines, not recipes that should be strictly followed. Hence, I could always find ways to adapt a particular pattern to the situation at hand.

What I like the most about them is the way they communicate experience. In particular, how experienced developers approached a particular problem, and why. I think that’s a great way to learn from others.

On that note, you might find this book useful: " Functional Programming Patterns in Scala and Clojure". I haven’t read it yet, but I find the theme very interesting. From the book description:

For each pattern, you’ll first see the traditional object-oriented solution, and then dig into the functional replacements in both Scala and Clojure. These patterns are common in the functional world and deserve to become part of your problem-solving toolkit. On the object-oriented side, you’ll see many common patterns, such as Command, Strategy, and Null Object. On the functional side, you’ll learn core functional patterns such as Memoization, Lazy Sequence, and Tail Recursion.

Saludos!

1 Like

Ok, I feel I need to answer regarding this assumption people are making about me.

Patterns are pre-fabricated solutions extracted from the industry. In the case of the GoF, it took them years to collect all the patterns published.

If you jump into any problem expecting that a pre-fabricated set of solutions is going to be the perfect fit for everything, then you will suffer. However, if you go on with the mindset of re-inventing the wheel, you will be in a world of pain as well.

Obviously, in the end, the “it depends” meme applies to a lot, but you will eventually have to make decisions and having a set of pre-fabricated solutions that you can adapt to your problems is a very useful tool.

Most Seniors know this, and I personally defend that if you just apply patterns right and left without considering their cost, then you are not really a Senior. As Zack says in his mailing lists: “If you don’t know the downsides of the tools you are using, then you don’t really know the tools you are using” (sorry for bad paraphrasing, but this is the conclusion).

Thanks for the book recommendation @Cruz , but I have another one in mind (Functional patterns in F#) :stuck_out_tongue:

To what extent?

One issue with a large part of the readership of Design Patterns: Elements of Reusable Object-Oriented Software is selective reading which focused on applying patterns and not much else.

Seems a lot of people were possibly skipping the Intent and Motivation sections and more importantly the Applicability and Consequences (benefits and liabilities, tradeoffs).

Also a significant value of patterns is to serve as a vocabulary for more precise and effective communication. It makes it possible to reference the pattern in the context of a solution without having to go into nitty gritty detail (repeatedly).

Even with SOLID - “principles” may be too strong, “guidelines” would be more appropriate. And even those guidelines are not context-free. Following those guidelines still makes certain tradeoffs - tradeoffs that may be acceptable in many circumstances, not so much in others.

It’s always important to consider context and understand the tradeoffs.

While you may not agree with every aspect of The SOLID Design Principles Deconstructed (2013) it is interesting to see them challenged (if you’re still following the SOLID principles and you regard yourself as a competent developer, you need to start giving them up).

An earlier discussion - applying SRP to Elixir:

1 Like

I actually saw that talk a long time ago and it was illuminating. I actually do agree with him in a lot of keypoints :smiley:

Honestly, this is the most important bit in the whole thread for me. Reading (and listening!) with an open mind is an incredibly valuable skill. May it serve you well!

2 Likes

Erlang Master Class 1: Video 8 - Discussion

So you likely got sucked in by the functional (yet sequential) programming part and now are consciously perceiving the third hurdle: thinking concurrently.

I also think there is a fourth hurdle: supervision - i.e. designing effective supervision trees.

1 Like

When I started the book I had a copy of the GoF design patterns right alongside it. I still do, it’s a book I cherish a lot. But I now understand that the Elixir best characteristics are better fostered in a environment not restricted by the patterns of 15 years ago, made for C++.

Just to add some more history: Peter Norvig put together a nice analysis of the GoF patterns from the point of view of a Lisp programmer (http://norvig.com/design-patterns/design-patterns.pdf). This was in 1996 and revised in 1998. The short summary: In most cases, you don’t need a pattern, you simply use a function or closure. In Elixir, you will find exactly the same situation, including Lisp macros. Don’t try to use OO patterns in a functional programming language…

In a way, this is both humbling and freeing. Humbling because I have cemented my position as a beginner. It somehow feels like after years of being a Senior in other areas where I had nothing more to really learn, coming to Elixir is almost like going back to level 1.

Yes, this is the most shocking experience when changing your programming paradigms. I still remember the long time of confusion around 1990 until I understood OO coming from a Pascal background. Similar happened again when changing towards functional / actor-style programming (do you already know about functors and monads :wink:?) And I am pretty convinced that the same will happen when I dive into logic programming, may it be classical prolog or miniKanren. Take a look into the wonderful book “Structure and Interpretation of Computer Programs” from Abelson and Sussman on how to implement all these programming concepts in a functional setting.

4 Likes