DDD/CQRS/ES/NoSQL and functional programming

I was wondering if anyone is using Domain Driven Design and/or CQRS and/or Event Sourcing and/or NoSQL storage for a business application that is written in a functional language (in particular Elixir and/or Phoenix).

Hit or miss? I am getting some mixed signals. Any thoughts would be appreciated.

What is it?

Domain Driven Design

CQRS, Task Based UIs, Event Sourcing agh! - Greg Young
http://codebetter.com/gregyoung/2010/02/16/cqrs-task-based-uis-event-sourcing-agh/

CQRS - Martin Fowler

Event Sourcing - Martin Fowler

Some links…

Functional Data - Greg Young
Event Sourcing is “Functional Data Storage”

A Functional Foundation for CQRS/ES - Mathias Verreas

Domain Driven Design with the F# Type System - Scott Wlaschin

CQRS with Elixir and Phoenix - Jean-François Cloutier
http://jfcloutier.github.io/jekyll/update/2015/11/04/cqrs_elixir_phoenix.html

Building a CQRS/ES web application in Elixir using Phoenix
Case study describing how I built a web app following a Command Query Responsibility Segregation and event sourcing pattern.

14 Likes

I am using CQRS in one project. So this is qorking, and I see no reason why it would not.

CQRS/DDD and friends are not bound to object oriented programming at all. Also, Elixir/Erlang are not that much different from object oriented languages when you think about it.

What I am doing at the moment is that I have two supervised workers, started when my application starts:

  • CQRS.Commands
  • CQRS.Events

there’s little bit of DSL in CQRS.Commands to define the list of commands my system accepts. These look like:

public MyApp.LogIn   
public MyApp.Register
private MyApp.CreatePost
private MyApp.DeletePost`

I have a single controller exposed on /api/commands that accepts params that have command name and payload. Public commands are ones that do not require User fetched (have plug for that). Private commands are the ones that get forwarded payload, but additionally current user.

Command looks like this inside:

defmodule MyApp.Commands.Register do
  alias MyApp.Repo
  alias MyApp.User
  alias MyApp.Events

  import  MyApp.Validator

  alias MyAppCommands.Register.Form

  def execute(params, _ \\ nil) do
    form = Form.from_params(params)

    if valid?(form) do
      Events.trigger("UserRegistered", event_data(form))

      {:ok, %{message: "email conrirmation needed"}}
    else
      {:error, errors(form)}
    end
  end

  defp event_data(form) do
    %{
      id: UUID.uuid4(),
      first_name: form.first_name,
      last_name: form.last_name,
      email: form.email,
      lang: form.lang,
      password_hash: Comeonin.Bcrypt.hashpwsalt(form.password),
      email_confirmation_token: Ecto.UUID.generate
    }
  end

  defp token_for_user(user) do
    Phoenix.Token.sign MyApp.Endpoint, "user_id", user.id
  end
end

and in my system commands can either succeed :ok , then generate some events, or fail (return tuple of :error and validation messages.

I do simple validations with https://github.com/CargoSense/vex.

Generated events are being dispatched by my CQRS.Events, that is also a bit of DSL that lists all the events that can happen. They return nothing and they should succeed. I do not have the strategy worked out yet on what to do when they fail. I am experimenting here, not everything is yet perfect.

But basically, commands gets executed, generates some events, those events are handled by events handlers. This is all happening in two workers that are GenServers, so one thing at a time.

In addiion to that I package all my queries into MyApp.Queries… etc. Events modify data, queries read data, I have clear separation here.

Queries are executed normally in Phoenix controllers.

I use Phoenix’ views to serialize results of my queries and send over to client either via JSON or just embed the thing into conn and pass to view.

I haven’t figured everything out yet. I will wrap it up in a library, and write a blog post soon. Maybe it’ll be useful for someone and maybe I get some good feedback. I am not CQRS/ddd expert, so will need that.

6 Likes

A Whole System Based on Event Sourcing is an Anti-Pattern

Building Microservices with Event Sourcing and CQRS (from Java world but good presantation :slight_smile: )

4 Likes

Agreed, you should be cautious of using it for a complete system. That is probably going to “over complicate” your application and have you think about things that should not be relevant.

I think one of the challenges in a traditional language is that you soon end up with a bunch of framework-ish code. Written by yourself or even worse a 3rd party. A framework that is probably not going to be a very good fit in the long run.

Using a functional language you don’t need to create this framework for it. You need things like functions, pattern matching, left fold and that’s already build-in. Process managers, actors, supervision, it is already there!

Hey, Greg Young isn’t wearing an Erlang T-shirt for nothing right?

2 Likes

I think if you use functional language or OO language, actor based system or micro services in all cases you can use CQRS or not. It depends of your domain problem you want to solve.

For example in Java/Scala when you use http://akka.io/ (actor based system) there is something called akka persitance where you can persists messages in actor queues. You have actor(s) who role is only persists other actor messages (in this way you don’t slower other actors work). There is also framework build based on this https://www.lightbend.com/lagom.

1 Like

Sure, it is possible.

The question -or my question, my reason for posting this topic- is if using a functional language makes some parts more trivial (like event sourcing) eliminating the need for a framework. Or am I overestimating the use of functional languages for this purpose? Is a functional language more of a natural fit?

Just touched functional languages so please be gentile.:slight_smile:

2 Likes

It certainly seems like Elixir is a good fit for DDD, CQRS, ES. mix can create umbrella app structure which is suitable for carving out isolated ‘apps’ or bounded contexts from a domain driven approach. I like the idea of starting by modeling the domain of what the core behavior is and then later adding phoenix as an app to umbrella when I get to it. Some say the folder structure and app/modules should be named the same as the domain ubiquitous language so that one can look at your folder and know the core behavior without reading all the code. The umbrella app has the potential of being much more framework agnostic then say starting out with a rails new or phoenix new skeleton which may skew one to start with persistence, mvc or scaffold first development, none of which are your domain.

ES and CQRS talk about using events to update projections for read models, and this seems like a good match to Elixir too since Elixir/Erlang is all about processes and message passing.

As to whether or not functional programming makes things easier or not, I’m not sure. I think it is a toss up for me. I’m more used to OO from Ruby but I can definitely see the benefit of not hiding state in objects and having to pass data explicitly everywhere. That might make it easier to reason about things.

I’m attracted to the ideas behind DDD but I don’t know where to start, what would the implementation look like?

Suppose I have thousands of merchants. Merchants have orders and inventory. I’m trying to think through DDD and designing with bounded contexts. Such that when an order is imported maybe there is a process that emits an OrderImportedEvent. And an interested party would subscribe to this event and update it’s own models it’s interested in. For example Inventory process might subscribe to the OrderImportedEvent and decrement it’s quantity available. I understand you can pass specific messages to specific processes with PID. But can I have events emitted into some event bus where multiple interested parties are subscribed to it? Merchants have private tenancy… meaning there is no shared data between merchants. So there is no reason for one merchant to be listening to another merchant’s events. Is each merchant’s bounded context or maybe event each aggregate a long running process that is waiting for messages?

Now that I’m writing this out… I just thought of an idea: Maybe each merchant gets their own supervised process tree for each aggregate like Order, Inventory, Product etc. Within the process tree one of the supervised processes is also a GenEvent manager (one per merchant). And all the aggregates add handlers for events that they want to listen to. Does this make sense as an approach? I have also read that GenEvent has issues, isn’t used much, and there are plans to replace it. Any thoughts? What should I use instead?

3 Likes

Another interesting presentation (cqrs etc. with erlang):

1 Like

I’m just getting started with Elixir, but have many years experience of DDD et al. Here are some thoughts on how I currently see DDD and CQRS patterns and concepts resonate with Elixir and the OTP.

This post assumes that you are familiar with the terminology from Evans book: Domain-Driven Design: Tackling Complexity in the Heart of Software

Entities and Value Objects are straight up maps or structs in Elixir. Obviously entitiy-structs are not mutable, but different structs with the same Id (intrinsic property of entities) can represent the entity at different points in time.

Aggregates could be implemented as GenServers. The server would serialize writes and validate business rules. In a simple (non CQRS) setup you could read by calling the GenServer. With the downside that reads and writes would not be concurrent.

Repositories would create and supervise the aggregate servers and handle communication with a storage for persisting their state upon request.

Services (as of the Blue book) are just functions operating on the domain by sending messages to the aggregate servers.

A slightly more involved aggregate implementation approach, with better performance would go the CQRS (without ES) route. It could let the read side mirror the aggregates of the write side, and let the aggregate servers only provide a command interface. The read side could be implemented as a ETS table owned by the repository and populated by aggregate servers after successful writes.

Expanding this sketch for a full out CQRS with ES solution, with denormalized read models etc. would probably be straight forward. Let the aggregate servers push events to an event store server and let that server update the read models in the ETS table.

Disclaimer: I’m new to Elixir and haven’t built any system along these lines yet.

2 Likes

I would definitely agree with @anders in your mappings to functional constructs. Perhaps the only one I am not sure about is services. I think that services should be processes that in most cases are an injected dependency into any operating code.

I previously wrote quite a bit about ddd in ruby and am still in the process of experimenting with the concepts in elixir.

In fact I think that the running Todo example from “Elixir in Action” by @sasajuric looks a lot like DDD and CQRS-light even though the terminology from Evans book is not used “strictly”. At least that’s how I read it.

@Crowdhailer: Regarding the services. This is the thing I too am least sure about. In OO you need the awkward verb-as-noun thing. It doesn’t really make sense here. Instead I see it as just functions. But they are functions within a domain (with names from the ubiquitous language and all the rest). So, not all functions in the application are domain functions. Hope that makes sense :slight_smile:

And you also have your long running Sagas. Which I probably would implement as GenServers. But that’s another story…

It’s certainly not intentional :slight_smile:
That said, I’ve read the Evans book a long time (~ 10 years) ago, so it’s possible that some ideas got stuck in my subconsciousness and are still present in my coding style :slight_smile:

5 Likes

When Chris and I we were discussing the changes upcoming to Phoenix 1.3 generators, we both agreed that we wouldn’t want to force commands/operations at the module level. Your commands should be functions. So I would rather attempt something such as:

public &MyApp.Accounts.login/2
public &MyApp.Accounts.register/2
private &MyApp.Posts.create/2
private &MyApp.Posts.delete/2

I can easily see functionality being shared between those commands. For example, both create and delete command for posts likely share similar authorization rules. So breaking it at the module level seems to impose the wrong level of granularity.

2 Likes

OK @josevalim, I think there is some definitions mismatch. In DDD/CQRS we have a command, which has a name, data, possibly also ID and timestamp (depending on implementation), and a command handler.

So a command, quite naturally fit into a module and could be:

%TransferMoney{from: “SOME IBAN”, to: “SOME IBAN”, amount: 1234)

and the command handler will be a function.

BTW: Have a look at this project on GitHub: https://github.com/slashdotdash/commanded that is pretty young but I did give it a shot yesterday and it seems to work surprisingly well and has some advanced features.

4 Likes

That clarifies it, thank you!

1 Like

####CQRS - Apache Kafka

Stream processing, Event sourcing, Reactive, CEP… and making sense of it all
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?

“Actor frameworks such as Akka, Orleans and Erlang OTP are also based on streams of immutable events. However, they are primarily a mechanism for concurrency, less a mechanism for data management. Some actor frameworks do have a distributed component, so you could build a distributed stream processing framework on top of actors. However, it’s worth looking carefully at the fault-tolerance guarantees and failure modes of these systems: many don’t provide durability, for example.”

True for Erlang/Elixir?

Some Erlang/Elixir libraries for Apache Kafka…


https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-Erlang -> https://github.com/klarna/brod

I would agree that Actor model is type of concurrency model. The higher abstract level are stream processing models.

In Scala word there is streaming frameworks like Akka Streams. There are some discussion that actor model is low level and does not compose well and better is using streaming model.
Which shoe fits you? Comparing Akka Streams, Actors, and Plain Futures.
Actor message passing is the GOTO of concurrency. Compose with futures instead.

For elixir I would say something equivalent is GenStage and Flow
But I have not using it, so maybe somebody can confirm if we can comprare GenStage to stream processing.

PS
I don’t want start any holy war :slight_smile:

1 Like

Based on this video and the code posted by the presenter, I published a small boilerplate: https://github.com/work-capital/elixir-cqrs-eventsourcing , but I admit that the approach used by https://github.com/slashdotdash/commanded is more clear and easier to debug and test. My question is Postgre VS Eventstore.