Hexagonal architecture in elixir

[anthonator]

@Apemb I’ve tried hexagonal implementations before and working with the Ecto/database presented challenges. I felt like I had to either a) implement an interface on top of Ecto to query data in a way that doesn’t couple Ecto to the core or b) create functions for every piece of data I needed. We also never really figured out a good way to handle transactions. Curious if you have any experience/thoughts on this.

In more recent hexagonal-ish builds I opted to use Ecto directly in the core and deal with the testing/coupling consequences.

[Apemb]

Oh well, that one was not easy…

The way we did it was to use a DDD tactical design pattern : the aggregate. Long story short, it is (as the name suggests) an aggregate of multiple structs, that actually “change as one”. There is quite a bit to the story, but I feel it is not the space to discuss that, and I am not really an expert on DDD. I could find links for you to look into DDD - I wrote down two books at the end.

The interesting part is that this big struct, the aggregate, is part of the domain, and is designed to represent one business transaction that is either fully successful or a complete failure. A business transaction is one aggregate change. When given to save to the aggregateRepository module, that aggregate is transformed into a changeset spawning multiple tables. That is the way we made it work. No Ecto leakage outside the Infra module, the transaction is that changeset.

So the problem in the infrastructure is solved by a more complex design of the domain struct, and as they are quite complex and do not look much like what is the database, have adapters that do have a bit of logic - transforming the tables into the aggregate is easy, transforming the aggregate into a changeset that is not fun… I think we would do a different and much simpler database schema if we had to do it again.
Those aggregates are also the reason why the query part of the application came to life, as manipulating those big aggregates for a small read was increasingly painful performance wise. (Some of our aggregates needed a join query of more than 10 different tables…)

Yet again, I think it was worth it, because following the DDD teachings is about more than technicalities, and more about a better way to communicate with the business persons the team works with. The aggregate reacts like the concept the stakeholder say its business concept does. It has the same name, works the same way. The price you have to pay is plumbing, and in our case a read part of the application.

I encourage you to read Eric Evans’ “Domain-Driven Design: Tackling Complexity in the Heart of Software” Book or the shorter “Domain Driven Design Distilled” by Vaughn Vernon. (I read the short one, and I should do it again because I feel I missed some parts). Either one will help you understand the philosophy behind the aggregate

Sorry to answer your question by a concept that takes a book to understand, if I knew how to explain it better I would, but really I am not yet able to summarise DDD in one blog post ^^. Maybe in a year or two.

[dimitarvp]

Quite interesting all of that.

I was about to suggest you use Ecto.Multi creating functions to actually have an Ecto-backed business transactions but then I saw you wanted them to have module/function names that correspond to what the business stakeholders call these business operations so I guess that wouldn’t be a good suggestion.

[LostKobrakai]

@anthonator

In more recent hexagonal-ish builds I opted to use Ecto directly in the core and deal with the testing/coupling consequences.

To be fair it depends on what you don’t want to be coupled to. If it’s “the DB” then MyApp.Repo is an implementation of the Ecto.Repo behaviour already and therefore decoupled. You can implement non db based/mocked repos. The biggest complexity with that approach are likely the functions arguments you need to handle – especially anything Ecto.Query. They can’t be well inspected and be acted upon without having a proper db (sql) to handle them. Thankfully almost all functions a repo has to deal with need Ecto.Queryables and not Ecto.Querys. So your core could build domain specific structs, which are queryables, which are expectable/testable, and only the protocol implementation deals with converting those structs into actual Ecto.Querys, which db based Repos use.

@anthonator

We also never really figured out a good way to handle transactions.

This is where Ecto.Multi becomes useful. It allows you to wrap business operations into a structure you can combine for transactional consistancy.

All in all it should be possible to build a project, which applies the ideas of hexagonal architecture, where the db and querying it is not part of the core, but you can code the core using most of ecto’s api.

It becomes a whole lot more work if you want do be decoupled from ecto though.

[LostKobrakai]

I just tried that out and tbh I don’t even think it’s that bad:

LostKobrakai/testing_ecto

I mean I’m not sure how bad the mocking might become in a more complex setting, but the sql stuff is cleanly separated from the business logic. Might do this more often from now on.

[anthonator]

@LostKobrakai

It becomes a whole lot more work if you want do be decoupled from ecto though.

Yes, this is where it gets tricky. @Apemb above mentioned no Ecto leakage outside of the infrastructure portion of the application. This is similar to what we did. We never found a good way to take advantage of the power of Ecto’s API in this situation (e.g. Ecto.Multi).

What you described is more or less how I’ve been doing it recently. At a certain application/team size I think abstracting out the database can be a fool’s errand. I always create a nice public interface with domain structs that are used to communicate with the outside world (e.g. Phoenix) but I don’t shy away from using Ecto in the application core. I just haven’t found it to be worth it. I do abstract out calls to 3rd party API’s using a port/adapter pattern.

[LostKobrakai]

@anthonator

Yes, this is where it gets tricky. @Apemb above mentioned no Ecto leakage outside of the infrastructure portion of the application. This is similar to what we did. We never found a good way to take advantage of the power of Ecto’s API in this situation (e.g. Ecto.Multi ).

“no Ecto leakage outside of the infrastructure” and “We never found a good way to take advantage of the power of Ecto’s API” sound like a contradition. How would you expect to use ecto api, but not “leak ecto”?

@anthonator

At a certain application/team size I think abstracting out the database can be a fool’s errand.

Abstracting the actual database integration away is a good thing. Abstracting knowledge about persistance away I’d say is a fool’s errand. Especially for all those webapps, which are essentially just glorious excel sheets. It makes no sense to have the core of an application be unaware of persistance, when all it does is manage data going in/out of it.

[anthonator]

@LostKobrakai

“no Ecto leakage outside of the infrastructure” and “We never found a good way to take advantage of the power of Ecto’s API” sound like a contradition. How would you expect to use ecto api, but not “leak ecto”?

Right, siloing Ecto into the infrastructure portion of your application makes it difficult to take advantage of Ecto’s powerful API.

@LostKobrakai

Abstracting knowledge about persistance away I’d say is a fool’s errand.

Yes, this is what I meant.

[al2o3cr]

@LostKobrakai

“no Ecto leakage outside of the infrastructure” and “We never found a good way to take advantage of the power of Ecto’s API” sound like a contradition. How would you expect to use ecto api, but not “leak ecto”?

I’ve seen those goals accomplished simultaneously in Ruby before - it led to a base layer of ActiveRecord models (with a scorching case of AnemicDomainModel because the classes were glorified DTOs), an intermediate layer that mostly just copied ActiveRecord data into plain Ruby objects because Abstraction, and then an ever-widening layer of module functions on top that used AR functions to get results and then translate them to POROs.

The phrase “in case we want to swap out ActiveRecord” was said at least five times a day.

IMO there are two overlapping reasons to not worry so much about “wrapping Ecto”: if the application controls the database (via migrations etc) it’s not really the same thing as the “external dependencies” that need decoupling against unexpected changes, and Ecto is already essentially an “adapter pattern” on top of the raw database layer.

[dimitarvp]

Reading @al2o3cr and @LostKobrakai made me think that in the rare cases where I care about isolation I just try and make sure that I don’t depend on stuff provided by ecto_sql (and even that can be very hard as @LostKobrakai pointed out). Depending on ecto itself is completely fine – because it does not deal with a specific database.

Leak-less isolation is IMO a pipe dream. Sure it might accelerate your tests but at what cost? Eventually you’ll fail accounting for something that only a live database does and then your tests become more or less worthless. Which is what a lot of unit tests out there are anyway – just a wishful thinking of the dev.

As for the so-called business transactions… meh. Just use Ecto.Multi and stop sweating about it. Ecto is a fantastic and well-tested library. If you go to such levels of paranoia you might as well ditch the interpreted language and VM concepts altogether and go lower-level: to C++, D, Rust, OCaml, Nim, Haskell etc.

@al2o3cr

The phrase “in case we want to swap out ActiveRecord” was said at least five times a day.

LOL, hits too close to home for the teams I’ve worked with in my six years of Rails work. People just love to pretend that they can swap any part of their app anytime they want – including when this part is a hugely important piece without which most of their app wouldn’t be anything more than a student exercise (since, you know, real-world apps want to persist data).

I get the motivation but IMO a lot of teams get carried away and forget where to draw the line.

@LostKobrakai

Especially for all those webapps, which are essentially just glorious excel sheets.

Now that’s a fact (it’s “glorified” btw, not “glorious”). I’d bet a few beers that a lot of apps out there can be rewritten in any language with an Airtable connector library and that they might work even better compared to the originals (and I include Phoenix app in this category as well).

Web frameworks are in general supposed to bring additional value over the basic “HTTP router to controller code with some views and templates sprinkled in” but many people stop at that point which turns into an epic fight that boils down to “how do we put this square peg into the round hole”.

As I get older I start finding myself thinking how do I make web apps with almost no code and just clever integration between several services. It’s possible, actually.

[LostKobrakai]

@dimitarvp

Web frameworks are in general supposed to bring additional value over the basic “HTTP router to controller code with some views and templates sprinkled in” …

To be fair this wasn’t actually meant as a bashing of simple web applications. Web applications can bring business value just by presenting data in a more practical way than a bunch of tabular sheets and without any fancy stuff on the backend. What I wanted to say is that in such cases people need to acknowledge the fact that their core business domain is moving data in and out of some database (airtable is just another db tbh) and there’s no sense in trying to abstract away the fact one is dealing with some kind of persistance.

[dimitarvp]

@LostKobrakai

To be fair this wasn’t actually meant as a bashing of simple web applications.

I know. Recently I kind of drift away and side-step topics. That’s on me.

Apologies for that but I still hope you don’t mind the side discussions every now and then.

@LostKobrakai

in such cases people need to acknowledge the fact that their core business domain is moving data in and out of some database

Yep, completely agreed on that. I also said above that I don’t get the huge efforts to pretend that we’ll never persist anything. And, tests are an additional code base that has to be supported which is a vastly underestimated development cost (and that code base is also usually more brittle than the code that it tests).

[Apemb]

I like this thread, and I agree with most of what I read.

But if you are talking about Hexagonal Architecture, you do have to abstract the database mapper layer. Or else you have a conceptual leak into your domain. The database and Ecto are flexible and powerful tools, with lots of possibilities, but they do force you do model things a certain way.

If you have simple responsibilities, and simple relationships between your objects you might not need the decoupling between Ecto and your Business Rules. If your Business Rules are not too complex, Phoenix Contexts are a good start, and do go a long way.

As a matter of fact, in my case, knowing all that, we did start with Ecto and basic Phoenix Contexts, and it proved to be not enough. The database modelling was too limiting, the business rules were too big for contexts, and testing started to become horrible (think inserting tens of objects every time because it was the only way to have a correct state in the database. We had factories, but not enough, not conviennent, and increasingly brittle…). So we chose to migrate to a more flexible architecture.

I do not suggest to anyone to start with a complex and over-engineered architecture like the Hexagonal, in elixir from the start. Indeed, as it was said, you most certainly do not need it. And if you do not it will slow you down. It is a compromise : do you win more time having the freedom to model your business structs and rules as you want, or do you loose more time writing converters between the layers ?

@dimitarvp

I also said above that I don’t get the huge efforts to pretend that we’ll never persist anything

That is not what you do, we had methods named quite explicitly save ou delete etc. But the persisting was juste not represented as manipulating Ecto, and not with Ecto.Structs. Because Ecto and Ecto struct limit your modelling and testing possibilities.

For context, we had to create a calendar (quite a simplification, it was not juste google calendar, but good it is a good enough analogy) but design an ecto struct that represent an infinite recurring series of events on flexible patterns was very, very painful. And that is why we changed and went for the hexagonal archi.

[LostKobrakai]

@Apemb

For context, we had to create a calendar (quite a simplification, it was not juste google calendar, but good it is a good enough analogy) but design an ecto struct that represent an infinite recurring series of events on flexible patterns was very, very painful. And that is why we changed and went for the hexagonal archi.

This sounds like a place where trying to map the domain model to the ecto model is no longer a lot of additional complexity, but complexity inherent to the problem domain. This is in my opinion much more worthwhile than trying to separate domain model from the ecto model, when both essentially look the same and one would effectively duplicated code.

[dimitarvp]

Likewise – I agree with most of what you said and I like your balanced approach!

One thing I’d grumble about if I was code-reviewing is this:

@Apemb

But the persisting was juste not represented as manipulating Ecto, and not with Ecto.Structs.

I am aware there are cases where Ecto structs aren’t enough but I remain sceptical as to how many projects actually need something outside of that. Modelling future datetimes is indeed a notoriously painful problem so you are likely correct that you need to step outside Ecto.

What I did in only one Elixir project ever was that we had one module with business context functions that tackled conversions exclusively: they received Ecto schema structs as input and returned plain Elixir structs as an output (and then vice versa, almost: the outputs were Ecto.Multi structs, not Ecto schema structs). And then all other business functions only operated on those plain Elixir structs.

Is that what you ended up doing (if I am reading you correctly)?

[Apemb]

Yes our ports return only KairosDomain plain structs. Structs that are some quite similar, some loosely and some quite different from the schema.

All the reste of the functions inside the command modules and domain modules do as if saving those struct was easy, use the correct port, and all is well.

The structs that end with Table are Ecto Structs, those with Entities are Domain Struct. (Our naming is like, Entites are readonly struct, Aggregates are the state-modifier structs)

# From KairosInfra.RecurringShiftEntityAdapter module

  alias KairosDomain.RecurringShiftEntity
  alias KairosInfra.RecurringShiftTable

  def from_recurring_shift_table(%RecurringShiftTable{} = recurring_shift_table) do
    %RecurringShiftEntity{
      id: recurring_shift_table.id,
      last_shift_generation_date: recurring_shift_table.last_shift_generation_date,
      timezone: recurring_shift_table.shift_service_request.warehouse.timezone,
      recurrence_pattern: adapt_to_recurrence_pattern(recurring_shift_table),
      recurring_template: adapt_to_recurring_template(recurring_shift_table)
    }
  end

# from KairosInfra.RecurringShiftEntityRepository

  def get(id) do
    result =
      RecurringShiftTable
      |> where(id: ^id)
      |> preload(
        shift_service_request: [warehouse: []],
        recurring_provider_assignments: []
      )
      |> Kairos.Repo.one()

    case result do
      nil ->
        {:error, {:resource_not_found, [name: :recurring_shift_entity, id: id]}}

      recurring_shift_table ->
        {:ok, RecurringShiftEntityAdapter.from_recurring_shift_table(recurring_shift_table)}
    end
  end

Those are one example with a loosely similar domain struct from ecto struct. Yes there is an adapter, yes the repository is not a plain Ecto function, but we gain a better error message, and writing the whole thing is like 5 min testing included. The longest part is creating the base data for the test factories. The rest is copy paste.

Why we did that on that particular struct, is more about consistency. We needed Ecto separation for one, we decided that it would be the way for all. (But as we refactored stuffed as we needed, I guess some stuff will never migrate as they never will be modified again…)

Is that needed for every project, no indeed, I do not think so either. In the startup I am working with right now, we have three different backends, and I see a moderate benefit in changing the architecture for one part of one of the three. What I am hopping is extracting that part into its separate app and going hexagonal on that one. And that isn’t very high on my to-do list

Hexagonal and clean archi are a nice clean way of organising code, extensible and easily maintainable, but expensive to migrate and in my opinion, useful only on a long and big project, or for complex business critical applications.