Community Context Exercise/Learning Discussion

It looks like you are trying to implement devise for phoenix.

While I understand such a need, I would mention that both Elixir and Devise are coming from plataformatec. So It should be very natural to have an easy port… but it’s not the case.

Porting any library from OOP to FP requires changing paradigm. And that is hard, hard as porting ruby gems to elixir hex.

Was this meant for a different thread? I’m simply talking about higher level concepts, not necessarily reimplementing any library. The APIs and boundaries are all hypothetical. If you’d like to take a stab at a different portion of the project unrelated to accounts, such as posts, I’d encourage you to do so.

2 Likes

In that particular case I was purely focused on determining where the responsibility for “composing the email” was - I wasn’t trying to determine a full picture of all the contexts involved - as a matter of fact the more I think about - even that URL/web page shouldn’t be the Account context’s responsibility - because that’s about “the Web” - not Accounts; contexts collect cohesive capabilities. You’re probably familiar with this quote:

Every block of stone has a statue inside it and it is the task of the sculptor to discover it. Michelangelo

Context’s within a domain are typically not discovered by some deterministic process - usually it takes some poking and prodding to find out what needs to stay together (high cohesion) and what should be separated (low coupling).

@gregvaughn

The “how would I organize this if I had to operate and sequence this functionality on the command line” is an incredibly useful thought experiment - trying to define small highly focused commands that can perform work based on their own autonomous data has the tendency to shake loose things that can be separated while highlighting what needs to stay together.

The “password reset” (actually I was more thinking of “forgot password” to be honest) I’m talking about is a user-scenario or use-case. It starts with someone clicking a button on a web page, resulting with a message in the users email, which contains a link to a web page that lets the user specify a new password. The Account context supports this scenario but it is only really interested in

  • Issuing some sort of perishable correlation ID that needs to be associated with the new password
  • Consuming the correlation ID together with the new password in order to verify that the correlation ID is valid and hasn’t expired - in which case it accepts the new password.

However there are many other parts of the use-case that need to be implemented - so a use case crosses many contexts. Setting up web pages with dynamically generated URLs, composing emails, sending emails aren’t in “Account’s” job description.

Ideally, at any point you should be able to change how the context works internally and provided you haven’t changed the interface its clients should not care. So “when something else” takes over “Account’s” responsibilities you should be able implement the pre-existing interface with a wrapper implementation and keep going - if changesets are part of your interface that isn’t going to happen because it is unlikely that the replacement supports changesets - as that technology is tied to Ecto.

I disagree - identifying areas of high cohesion and low coupling has always been a design goal even in monoliths to enhance maintainability (partially through replaceability). In fact it’s a strategy used when moving to microservices - first identify the boundaries within the monolith then refactor to reflect the boundaries before finally splitting off the microservice. The advantage in a microservice is that the boundary is physically enforced - in a monolith the boundary is largely conceptual and only maintained through developer discipline - violating context boundaries in a monolith is incredibly easy and often tempting (undoing the benefit of keeping it clean in the first place).

If you define a context per table you essentially just end up with Table Modules which is hardly an improvement over Rail’s Active Record.

  • The point is that there is always an upfront cost to loose coupling.
  • If loose coupling is applied in the wrong place (i.e. not at a “natural” boundary) then it’s going to keep costing without ever generating any return.
  • However loose coupling in the right place pays huge dividends in terms of maintainability. Typically it manifests itself in terms of replaceability - especially when multiple changes hit the same context - while the interface to the context manages to isolate the clients of the context from those changes.
  • Unbridled tight coupling (never-ending shortcuts) will leave you with the proverbial big ball of mud.

The wrong boundaries are just as bad as no (or too few, too large) boundaries. Finding the optimal boundaries is rarely a picnic - typically it requires that you understand the domain quite well - often better than the stakeholder, service owner, project/product champion - it is far from a cookie cutter affair.

A map is a very generic Elixir data structure - changeset is not - compare the functions that support Map vs Changeset - so a map is preferable - a struct is good as long as it reflects a domain concept - Changeset is all about functionality that Ecto supports but nothing directly domain related. However within the context you can use Changeset (as arguments/return values of private functions that support the public functions) as it is likely that the implementation of a context would be replaced wholesale anyway.

Yes, but an error tuple is idiomatic in Erlang and therefore Elixir - so it is the context implementation’s responsibility to extract the error message from the Changeset and to wrap it in an error tuple before returning it to a client

My argument is that if it’s necessary to expose Changeset you’ve either

  • chosen the wrong boundary or
  • implemented a poor interface along the boundary
    .
4 Likes

I’m still mulling over the rest of your post, so I won’t be commenting on that until I feel I have a better understanding.

However, I’m not sure I’m convinced by your arguments concerning changesets. Let’s assume the following is returned by a module when a validation error occurs: {:error, changeset}. While it’s true that the Ecto.Changeset struct is part of the Ecto suite, it doesn’t change the fact that it is simply a bare map underneath, meaning all of the functions in the Map module can be used on a changeset. You could ignore the __struct__ field entirely or even convert it using Map.from_struct() before returning it. I’d argue that the data returned via a changeset is valuable regardless of whether not Ecto is being used within a project. Any error structure you come up with in your own application is very likely to mimic or copy the information already being returned by a changeset. You could simply extract the errors field and return that in your error tuple instead, but then you’d be missing out on a plethora of other useful information.

  • Ecto.Changeset is part of Ecto. No Ecto - no Ecto.ChangeSet.
  • Ecto is an implementation detail (related to the particular persistent storage technology that you are currently using), %Ecto.Changeset{} exposes a dependency to that implementation detail.
  • Dependencies on implementation details are a form of tight coupling.
  • Tight coupling between capabilities within the same context is OK.
  • Tight coupling across context boundaries compromises context integrity and autonomy as its clients will become coupled to its implementation details, creating very real obstacles for the internals of the context to evolve in the future because most changes will ripple to it’s clients (rendering the existence of the context kind off pointless).

The other issue is: valuable to who, exactly?

From Ecto github repository

  # If a new field is added here, def merge must be adapted
  defstruct valid?: false, data: nil, params: nil, changes: %{}, repo: nil,
            errors: [], validations: [], required: [], prepare: [],
            constraints: [], filters: %{}, action: nil, types: nil,
            empty_values: @empty_values

  @type t :: %Changeset{valid?: boolean(),
                        repo: atom | nil,
                        data: Ecto.Schema.t | map | nil,
                        params: %{String.t => term} | nil,
                        changes: %{atom => term},
                        required: [atom],
                        prepare: [(t -> t)],
                        errors: [{atom, error}],
                        constraints: [constraint],
                        validations: Keyword.t,
                        filters: %{atom => term},
                        action: action,
                        types: nil | %{atom => Ecto.Type.t}}

Now for most of this discussion I’ve been assuming that we are talking about a domain context. Very little of the above information is of any interest to domain logic. domain logic cares about domain types. domain types are supposed to strive towards Persistence Ignorance (PI). This typically translates to going through considerable lengths to hide the persistence mechanism from the client of the context. This could mean:

  • Transferring the data contained in an Ecto.Changeset to “Plain-Old-Data” (from POJO and POCO; “Plain-Old-Java-Object” and “Plain-Old-CLR-Object” respectively) before returning it to a client. So if you return a map each key would relate to a domain concept and each value would either be a basic Elixir data type representing a domain quantity or an instance of a domain type. Essentially the Ecto schema structs act as mere “data-transfer-structures” (DTO).

  • Burying the N-PI (non-persistence ignorance) part of the data in a deep, dark, opaque corner of the domain type and banning the client from any direct access to type’s data requiring that all accesses have to go through domain module functions which of course know how to navigate the twisted internals of that particular type. It should be obvious that this approach is only a last resort; for pure results the former approach is preferred. Other than that “Ask for help, not information” is the module/interface design mantra that is used to try to avoid exposing types where it is necessary to track a dirty state (again “state” is the complexity culprit).

  • Ideally whoever “owns” the Changeset could hold on to it until (or re-retrieve it when) the “new versions” of the domain type instances “come back” and then use some kind diff-ing mechanism to generate the Changeset needed for Ecto. I’m fully aware of how redundant that sounds but to achieve a high level of PI, Ecto doesn’t go far enough for decoupling because it assumes that the logic that makes the changes also has the Changeset. However the domain logic is only concerned with creating the new, updated instance of the domain type, not knowing how to deal with Changesets - that would be considered accidental complexity that obscures the domain logic.

In the other post I stated this

However within the context you can use Changeset

In terms of DDD I was actually taking a lenient position - DDD actually wouldn’t even let something like a Changeset exist in a domain context. Within a subdomain a repository has the responsibility of dealing with persistent storage concerns - which includes all the data query language assets - but the repository is also only allowed to serve (and accept) “plain-old-data” and domain types to it’s clients. Which makes sense as that, for example, keeps the burden of handling the Ecto-related concerns solely with the repository so that the capabilities within the domain context are only dealing with domain related concerns - however if the context is small enough, the repository could be overkill.

And while on the topic of DDD - the Mailer wouldn’t be considered a context but a service:

A good SERVICE has three characteristics.

  1. The operation relates to a domain concept that is not a natural part of an ENTITY or VALUE OBJECT.
  2. The interface is defined in terms of other elements of the domain model.
  3. The operation is stateless.

Now I’m not here to serve out the DDD kool-aid because there are legitimate criticisms, for example:
Jim Coplien — Symmetry in Design
I think one of his implied concerns is that the third “D” is entirely focused on design, and that Domain Analysis (i.e. understanding the domain) seems to happen much too late in the process. Boundaries are often not chosen but discovered.

Patterns, Principles, and Practices of Domain-Driven Design, p.82

It’s important to be explicit about what context you’re using when talking with domain experts, because terminology can have different meanings in different contexts. As repeated throughout this chapter, multiple models will be at play in your domain. You need to enforce linguistic boundaries to protect the validity of a domain term. Therefore, linguistic boundaries are bounded context boundaries. If the concept of a product has multiple meanings inside the same model, then the model should be split into at least two bounded contexts, each having a single definition of the product concept.

Furthermore, I doubt that the Phoenix team was intending to push DDD as such. A Phoenix context:

  • Is about collecting capabilities exhibiting high cohesion in a single place (where tight coupling is OK)
  • Sets a boundary around these capabilities. The boundary is about loose coupling (the opposite of tight coupling) towards the context’s clients - largely to keep clients isolated from any changes internal to the context - up to and including the wholesale replacement of the context’s internals.

To reiterate:

  • Ecto and any associated data types are an implementation detail.
  • Letting implementation details cross the boundary leads to tight coupling and severely weakens the benefit of maintaining a context (and its boundary).

In any real-world application, you need a way to return validation errors to the users. Do you create a separate structure for each entity you have? This might be “correct” and “pure”, but it is definitely not practical.

While changeset implies there’s some data storage behind, it completely does not carry information what that data storage is. It might as well be an in-memory adapter. How deep does one go pruning structures from libraries? At some point, it’s necessary to decide something is a “core” library that is “safe” to use. Otherwise, it’s the same as falling deep into the NIH syndrome. Would anything change if the changeset code was copied into the application and called “MyApp.Changeset”? Would that mean separating from the database, would that be “correct”? For me, it wouldn’t change much.

In much of the DDD code I saw, validation is handled using exceptions. I generally find those to be a horrible choice to use exceptions (or any non-local code construct) for control flow. I’d take a plain-data changeset any time of day.

Phoenix contexts and DDD are related. They have similar goals - make you think about your domain first. But they are definitely not one and the same. One is purely a code organisation construct, another one a whole philosophy of doing software development.

2 Likes

Boundaries are about isolation with the intent of managing dependencies - so its always about having as few dependencies as possible and you want to be especially independent of things that you may want to change in the future. You also want to prune dependencies that may impose changes on you.

You are also supposed to depend only on the part that you actually use - which usually involves writing a thin wrapper around it that exposes the functionality in the way you need to use it.

Got nothing to do with “Not-Invented-Here” - everything with maintaining architectural choice and options.

The most severe case being needing to use a different mapper. If that new mapper is EctoX then maybe I might be able to still use Ecto.Changeset while taking some performance hit or being denied some new features until I change over to EctoX.Changeset. But what if I want to switch to UnectoV? They’ll have UnectoV.Diffset which is incompatible with Ecto.Changeset. So I’m either stuck with Ecto even though I want to switch or I have to upgrade the entire application because, silly me, I let Ecto.Changeset leak and bleed all over the place. At least with a DDD repository approach the changes are contained to the replacement of the DDD repositories and my domain logic and non-persistent storage infrastructure services aren’t effected.

The idea is to stop conflating unrelated concerns (persistent storage and domain logic) and avoiding (vendor) lock-in. Meanwhile I can use all the Ecto specific features I want in the DDD repository because all of that will have to be replaced anyway. And ultimately it’s about daring to judiciously invest in “pay now, save later” (though there are no guaranties in life) rather than the always easy “save now, pay a tons more later but who cares I’m not going to be around anyway” approach.

No, because MyApp.Changeset still serves the needs of the persistence technology (on Ecto’s terms) - not the domain proper. Renaming things does not reduce coupling.

For the domain an error tuple is quite sufficient. Furthermore if the error is related to Ecto or storage, log the details; the domain doesn’t need to know the gory details - just like you don’t spill the goods to the user on an “500 Internal Server Error”.

Just to be clear - it was never my intention to rip on Ecto - though sometimes I may not agree with how it’s employed. But I was stating that in my view there is a fundamental design tension between wanting to establish a boundary around a context while at the same time passing implementation revealing types like Ecto.Changeset through that boundary - and that doing so ultimately compromises the intent behind establishing the context boundary in the first place.

From that point of view stating that not passing Ecto.Changeset through that boundary “is definitely not practical” is to me tantamount to pouring gasoline on the fire of the “contexts are useless” camp.

1 Like

How would you choose to implement error handling for validations? While you’ve made your criticisms of passing changesets outside of a context known, I think seeing an actual example of what you’re proposing would benefit the discussion.

To get right to the point - if I’m investing in domain types I can’t use the validation functionality provided by Ecto.Changeset.

A major point of domaintypes is that instances will only contain information that is known to be valid in reference to the domain. Validation has to happen on data before it becomes a domain type instance. Data entering persistent storage should already be in the form of a domain type instance - i.e. it is already known to be valid. Data coming out of persistent storage should already be valid because it entered it as a domain type instance.

So I would say that Ecto.Changeset conflates the concerns of data mapping and data validation. Now that conflation is convenient for a "Phoenix is your application " development style - which is essentially what Eric Evans calls a “Smart UI”. In a Smart UI you don’t use domain types because the logic isn’t all that complex to begin with, so the overhead of developing a domain model isn’t warranted. The benefit of a Smart UI is that it is relatively easy to put together - even by a team who doesn’t necessarily have deep domain knowledge as there isn’t any complex domain logic to contend with - your typical (close to) CRUD style application.

The problem with a Smart UI is that there is very little margin for growth and development because any additional complexity will quickly push it towards a big ball of mud. Now if in a Smart UI there is a cause to create a context, I guess it would make sense to pass Ecto.Changeset through the boundary - because there aren’t any domain types in the application to begin with - the Ecto schema structs are the “acting domain types”. But if I found a justifiable reason to create a context within a Smart UI, I would get very, very deeply concerned because that would be a potential indicator that I chose the wrong style of application.

The “Phoenix is not your Application” development style is the game changer. “Functional Web Development with Elixir, OTP, and Phoenix” has been making headlines lately - lets look at that:

/_build/dev/lib/islands_engine/ebin/islands_engine.app

{application,islands_engine,
             [{registered,[]},
{description,"islands_engine"},
{vsn,"0.0.1"}, {modules,['Elixir.IslandsEngine','Elixir.IslandsEngine.Board',
                        'Elixir.IslandsEngine.Coordinate',
                        'Elixir.IslandsEngine.Game',
                        'Elixir.IslandsEngine.GameSupervisor',
                        'Elixir.IslandsEngine.Island',
                        'Elixir.IslandsEngine.IslandSet',
                        'Elixir.IslandsEngine.Player',
                        'Elixir.IslandsEngine.Rules']},
              {applications,[kernel,stdlib,elixir,logger]},
              {mod,{'Elixir.IslandsEngine',[]}}]}

With the exception of GameSupervisor all the OTP application modules Board, Coordinate, Game, Island, IslandSet, Player, and Rules deal with domain types and capabilities. No Changeset in sight.

“Foul” you cry - “that application doesn’t even store state in persistent storage - all state is stored in process state!”

So???

  • What business of the client (to the application) is it, how state is stored within the application?
  • Would the domain types or capabilities exposed change if some of the state was stored in ETS tables?
  • What if some of the state was stored in mnesia/amnesia?
  • What if we used plain-old Postgrex (i.e. no Ecto)?

So why would I start leaking Changesets the moment I use Ecto?


The Smart UI “Anti-Pattern”

. . . That sums up the widely accepted Layered Architecture pattern for object applications. But this separation of UI, application, and domain is so often attempted and so seldom accomplished that its negation deserves a discussion in its own right.

Many software projects do take and should continue to take a much less sophisticated design approach that I call the Smart UI. But Smart UI is an alternate, mutually exclusive fork in the road, incompatible with the approach of domain-driven design. If that road is taken, most of what is in this book is not applicable. My interest is in the situations where the Smart UI does not apply, which is why I call it, with tongue in cheek, an “anti-pattern.” Discussing it here provides a useful contrast and will help clarify the circumstances that justify the more difficult path taken in the rest of the book.

❊ ❊ ❊

A project needs to deliver simple functionality, dominated by data entry and display, with few business rules. Staff is not composed of advanced object modelers.

If an unsophisticated team with a simple project decides to try a Model-Driven Design with Layered Architecture, it will face a difficult learning curve. Team members will have to master complex new technologies and stumble through the process of learning object modeling (which is challenging, even with the help of this book!). The overhead of managing infrastructure and layers makes very simple tasks take longer. Simple projects come with short time lines and modest expectations. Long before the team completes the assigned task, much less demonstrates the exciting possibilities of its approach, the project will have been canceled.

Even if the team is given more time, the team members are likely to fail to master the techniques without expert help. And in the end, if they do surmount these challenges, they will have produced a simple system. Rich capabilities were never requested.

A more experienced team would not face the same trade-offs. Seasoned developers could flatten the learning curve and compress the time needed to manage the layers. Domain-driven design pays off best for ambitious projects, and it does require strong skills. Not all projects are ambitious. Not all project teams can muster those skills.

Therefore, when circumstances warrant:

Put all the business logic into the user interface. Chop the application into small functions and implement them as separate user interfaces, embedding the business rules into them. Use a relational database as a shared repository of the data. Use the most automated UI building and visual programming tools available.

Heresy! The gospel (as advocated everywhere, including elsewhere in this book) is that domain and UI should be separate. In fact, it is difficult to apply any of the methods discussed later in this book without that separation, and so this Smart UI can be considered an “anti-pattern” in the context of domain-driven design. Yet it is a legitimate pattern in some other contexts. In truth, there are advantages to the Smart UI, and there are situations where it works best—which partially accounts for why it is so common. Considering it here helps us understand why we need to separate application from domain and, importantly, when we might not want to.

Advantages

  • Productivity is high and immediate for simple applications.
  • Less capable developers can work this way with little training.
  • Even deficiencies in requirements analysis can be overcome by releasing a prototype to users and then quickly changing the product to fit their requests.
  • Applications are decoupled from each other, so that delivery schedules of small modules can be planned relatively accurately.
  • Expanding the system with additional, simple behavior can be easy.
  • Relational databases work well and provide integration at the data level.
  • 4GL tools work well.
  • When applications are handed off, maintenance programmers will be able to quickly redo portions they can’t figure out, because the effects of the changes should be localized to each particular UI.

Disadvantages

  • Integration of applications is difficult except through the database.
  • There is no reuse of behavior and no abstraction of the business problem. Business rules have to be duplicated in each operation to which they apply.
  • Rapid prototyping and iteration reach a natural limit because the lack of abstraction limits refactoring options.
  • Complexity buries you quickly, so the growth path is strictly toward additional simple applications. There is no graceful path to richer behavior.

If this pattern is applied consciously, a team can avoid taking on a great deal of overhead required by other approaches. It is a common mistake to undertake a sophisticated design approach that the team isn’t committed to carrying all the way through. Another common, costly mistake is to build a complex infrastructure and use industrial strength tools for a project that doesn’t need them.

Most flexible languages (such as Java) are overkill for these applications and will cost dearly. A 4GL-style tool is the way to go.

Remember, one of the consequences of this pattern is that you can’t migrate to another design approach except by replacing entire applications. Just using a general-purpose language such as Java won’t really put you in a position to later abandon the Smart UI, so if you’ve chosen that path, you should choose development tools geared to it. Don’t bother hedging your bet. Just using a flexible language doesn’t create a flexible system, but it may well produce an expensive one.

By the same token, a team committed to a Model-Driven Design needs to design that way from the outset. Of course, even experienced project teams with big ambitions have to start with simple functionality and work their way up through successive iterations. But those first tentative steps will be Model-Driven with an isolated domain layer, or the project will most likely be stuck with a Smart UI.

❊ ❊ ❊

The Smart UI is discussed only to clarify why and when a pattern such as Layered Architecture is needed in order to isolate a domain layer.
There are other solutions in between Smart UI and Layered Architecture. For example, Fowler (2003) describes the Transaction Script, which separates UI from application but does not provide for an object model. The bottom line is this: If the architecture isolates the domain-related code in a way that allows a cohesive domain design loosely coupled to the rest of the system, then that architecture can probably support domain-driven design.

Other development styles have their place, but you must accept varying limits on complexity and flexibility. Failing to decouple the domain design can really be disastrous in certain settings. If you have a complex application and are committing to Model-Driven Design, bite the bullet, get the necessary experts, and avoid the Smart UI.

Eric Evans (2003). Domain Driven Design: Tackling Complexity in the Heart of Software (pp.76-79). Boston, MA: Addison-Wesley.

1 Like

I’m note sure that answers my question. I understand that you don’t want to have changesets involved at this point. What I’m asking is what a realistic alternative for validation errors would look like. Let’s assume that you’re given a basic registration form containing name, email, and password. Then let’s assume you have the following validation rules:

  • name, email, and password are required
  • name cannot be more than 100 characters
  • email must contain “@”
  • email cannot be more than 254 characters
  • password must be at least 8 characters and no more than 100 characters

If a single validation rule is violated, what would your error response look like? If multiple validation rules are violated, what would your response look like?

1 Like

I really don’t know how this adds anything to the discussion at hand.

The lowest common denominator is to use regular expressions to validate the various entries. As to whether or not to just report the first or all violations is probably dependent on the circumstances. Also if an SPA is used, in-browser based validation is necessary for user experience but the server side functionality would never rely on that and it would always do its own duplicated validation - yes, that type of duplication is annoying and tedious but in the end, it’s a necessary evil unless you are using something like Clojure on the server and ClojureScript in the browser so that you can use the same validation on both ends.

However a quick google lead me to Vex so I would see if I could leverage that within the application on the server side - it seems to focus on doing one thing well - which I prefer over this:
just because you can doesn't mean you should

It adds to the discussion because I’m trying to understand your position beyond the criticism. You’ve made it clear that you don’t believe we should be leaking changesets. However, I would like to see what you would consider a reasonable alternative. I’m not asking how you would validate the data. I’m asking what you would return from a function assuming there was a validation error. As it stands, I still agree with @michalmuskala when it comes to using changesets as a simple data structure to describe validation errors.

The point of this thread was to bounce ideas back and forth about how to structure contexts with high-level examples. Examples are key for people like myself to understand concepts completely. I feel that this thread is on the verge of getting derailed. If you feel as though contexts should never return a changeset, then I could see it being relevant to this discussion, but I’d ask that you’d offer a reasonable alternative via an example rather describing architecture design practices. I don’t think I’m asking anything unreasonable.

To bring the thread back on topic, would you care to share how you could structure any context in this application? It doesn’t have to be a large one, but you seem to be well versed in the design pattern you’re proposing and I’m curious to see how you would approach contexts.

It is fair to say that changesets are part of ecto and that they are controlled by Ecto. On the other hand, recently, when explaining what Ecto is I tend to say it’s a “data modelling and database library”. The truth is, Ecto could be two separate libraries - schemas & changesets in one and repo & all the database stuff in the other. This separation is one of the things I’m actively exploring. I frequently use changesets even for things that have no database at all - they are a convenient abstraction for data processing even without a database.

One thing I’m missing in many DDD discussions is the value of common abstractions. Sure, it’s possible for each context to build its own way of returning errors. It might be fine if that’s formalised. It’s much worse if it isn’t. If they return some haphazardly created data structures that have no “formal” structure and can (and will) change - from my point of view that’s exactly what will be the effect of using “raw” tuples or maps. Using a changeset, even though it comes from a library, gives some common structure and formalisation of the return value. This is very valuable. Using libraries imposes constraints and that is sometimes a good thing.

And it’s true an experienced and disciplined team can create their own abstractions. They can build a protocol for extracting errors or they can build an in-application library for common error handling. It will be perfect and work exactly like they need it to. There are obvious tradeoffs related to this, though.

I was trying to more clearly understand your position on the need for structured error data - so I came across this: Error Handling in Elixir Libraries

One thing that occurred to me in the category of “actionable errors” is that quite frequently the effective action in response to the error is entirely independent of the error reason.

When I have just about enough time to drive to a meeting and the car fails to start then an error report as to whether the battery died due the overnight frost or because a racoon decided to make a snack of the wiring is entirely unhelpful in my objective of reaching the meeting on time - I have to face the fact that the car won’t get me there and quickly find an alternate mode of transportation in order to meet my immediate objective.

Detailed error information is important for error logging and consistent formatting is extremely helpful for mining the logs. Detailed and specific error messages are also essential in the UI in order to quickly and effectively direct the user towards corrective action (though in SPAs that logic is typically entirely contained within the browser). In most other cases I would expect that using ok/error tuples (with a reason string) in the role of an Either type is quite sufficient especially if the message is not intended for the user of the system (because then i18n could become an issue).

However I would likely augment their use with the approach described here in order to bypass the typical awkwardness associated with ok/error tuples.

1 Like

The problem with using plain strings for errors (or atoms, tuples, etc) is that they are not extensible. You can see this problem with Erlang - it’s widely known for poor error messages. Most functions in :ets or :crypto will raise :badarg as error. Because this is their interface, they can never make the errors more helpful without breaking backwards compatibility. Given how fundamental these modules are, it’s probably never going to happen.

On the other hand, if you look at Elixir errors which are structs, you can very easily add a field and improve the error message without breaking any contracts.

1 Like

I’m sensing an assumption that contexts are a result of design activities - I would challenge that assumption and counter that in fact they are usually a result of domain analysis activities, especially when it comes to the DDD bounded contexts.

This does not preclude the possibility of boundaries being only discovered when coding is already underway - sometimes the facts that influence such a discovery are buried so deep that an effort would be cancelled due to Analysis Paralysis long before these details are unearthed.

One thing from DDD that doesn’t get enough attention is the practice of ubiquitous language:

A language structured around the domain model and used by all team members to connect all the activities of the team with the software.

Developers and domain experts have to share the identical terminology and mental model - to the point that they have to agree on a ubiquitous language domain glossary which includes all the terms and the detailed explanations.

I was once in a situation where I was continuously compiling and updating a unified domain glossary after every meeting for some time after project inception in order to unify the communications between the various stakeholders and developers who previously where using divergent terminology for the similar concepts and similar terminology for divergent concepts. It was during the act of talking to the domain experts and formulating this ubiquitous language that most of the boundaries started to reveal themselves.

So while, for example, a concept like product seems straight forward enough the relevant information about it can vary considerably among the various contexts like:

  • Procurement
  • Inventory
  • Pricing
  • Fulfillment
  • Sales
  • Marketing

which can lead to each context having it’s own, different product type which are all correlated via a correlation ID. Furthermore the product type used internally inside the context for collaboration between the context’s capabilities will likely be much more rich and detailed than the one it shares with its clients in order to stop the clients from becoming coupled to the context’s implementation details.

Now when it comes to fine-grained contexts it’s still about low coupling and high cohesion.

|                     | Tight Coupling                             | Loose Coupling                       |
===========================================================================================================
| Referencing         | Point-to-Point                             | Via Mediator                         |
| Communication style | Synchronous                                | Asynchronous                         |
| Data model          | Common complex types                       | Simple common types only             |
| Type System         | Strong                                     | Weak                                 |
| Interaction Pattern | Navigate through complex nested structures | Data-centric, self-contained message |
| Control of Process  | Central control                            | Distributed control                  |
| Binding             | Statically                                 | Dynamically                          |
| Platform            | Strong Platform dependencies               | Platform independent                 |
| Transactionality    | 2PC (two-phase commit)                     | Compensation                         |
| Deployment          | Simultaneous                               | At different times                   |
| Versioning          | Explicit upgrades                          | Implicit upgrades                    |

Asynchronous communication typically refers to (events). So loose coupling rarely comes easily nor cheaply.

But at times reversing the direction of dependencies is good enough: Dependency Inversion Principle in the Wild.

1 Like

Yes - but at the same time I think the sometimes obsessive concern with extensibility and reuse is what has gotten the practice of OO into trouble.

Seems to be a direct consequence of Chapter 13: Programming Erlang 2e:

In Erlang we have a large number of processes at our disposal, so the failure of any individual process is not so important. We usually write only a small amount of defensive code and instead concentrate on writing corrective code.

The Erlang philosophy for building fault-tolerant software can be summed up in two easy-to-remember phrases: “Let some other process fix the error” and “Let it crash.”

This seems to give rise to a tenet of relying on a set of well-defined, pre-selected contingencies whenever a “you can’t get there from here” situation arises - rather than expending effort on trying to micro manage the error response based on a plethora of detailed error information that may or may not be available.

An error occurs:

  • let’s give up OR
  • let’s try again again later OR
  • let’s try something entirely different

You can’t get any more loosely coupled than simply responding to the presence of the error while not depending on any of the details about the error.

And once even a single client depends on that field you’ll break it when you need to take it away again or move it, i.e. by offering too much detail you are limiting your own potential for future change without breaking things outside of your own boundary. Too much detail can be just as damaging as too little - possibly more so.

1 Like

@peerreynders
What I’m missing from your remarks is a differentiation between errors a user cannot directly act on (your car example) and ones, where the user can indeed act on. If there are components of your system failing to work as expected you can easily return a generic error and log the rest. But if things depend on the user you might as well tell them what is wrong with their side of it and not just “sorry, couldn’t work with that, please fix it on your own”. E.g. I like it when my printer tells me about the empty paper cartridge without me having to diagnose, why it won’t print. Same for empty inks. And at best my printer should tell me both issues at once, because then I don’t have to walk to my printer twice.

Lets address this specific example first. One thing you should notice is the impact that the error had on your objective. You started:

  • I want to print this.

error happens

  • I want to fix the printer.

Printing something and diagnosing printer problems are separate concerns - as per ISP that would benefit from separate interfaces - irrespective of the fact that they relate to the same physical object in the tangible world.

Different clients have different needs but they should all aim to minimize dependencies - to the benefit of both parties, the clients and the provider.

Some clients just want to print - if it fails they either give up or just try again later in case the problem was fixed. These type of clients should only be dependent on the printing interface - and to keep things simple, errors should be kept simple; printing either works or it doesn’t.

In some cases the change in objective is permanent. A periodic job prints a work schedule on paper for a high-EM environment. The job can’t fix the printer but it can do it’s best to speed things along. Now becoming dependent on the diagnostic interface becomes a deliberate choice. Rather than just sending a “failed to print” email to the admin, relevant information is gathered from the diagnostic interface to be included in the email.

As far as design goes it would be just as legitimate to avoid the dependency on the diagnostic interface and decide that it is good enough to stick to “failed to print”. In most cases the operator has a separate detailed console or dashboard to the printer anyway (and has to go to the printer to actually fix it), so the effort of duplicating the printer status in the email could be viewed as a form of “waste for the sake perceived convenience”.

Finally the case that you probably had in mind was that the new objective is ancillary to the original objective. Again the diagnostic interface can be used to aggregate the necessary information. But you should’t blindly send all available information but instead a priori determine what information is actually relevant to the end user and just send that. (And your original objective is still nixed if it turns out that you didn’t stock up on ink/toner - but that’s on you).

Now developers like interfaces where their favourite IDE can just latch onto those capabilities so that they can just look at the list of what’s available, pick and choose stuff and move on. However if that diagnostic interface was designed to minimize dependencies there will likely be only one single capability - something that accepts a diagnostic query. The good news is that the intent is to support older queries indefinitely while adding more advanced capabilities later - but that proprietary query language is just a different form of tight coupling - so probably need to put a façade around that.

In the “real” world printing often goes to a queue. That’s nice and asynchronous - i.e. loosely coupled. But by its generic self there’s a problem - all you know is that the job went onto the queue - so there is no way to find out that it printed or whether there is a problem. So you need some sort of a backchannel to get that information back. But how much error information do you send back? All of it? There could be a lot. And the more you share the more likely it becomes that you’ll run into versioning problems over time for those who actually use it. And if you only send some of it, how is that configured? Centrally for all users? Individually for each request, i.e. “if there is an error then send me this information”. Again the solution is to keep errors as simple as possible and to add diagnostic capabilities (e.g. another queue) for those who actually need it.

The problem with harmonized types (in this case the error that is returned) is that complexity is inflicted unnecessarily on the simple cases. That is an acceptable tradeoff if only 5-15% of your cases are simplistic while the remainder is inherently complex. But when 85% of your cases are simplistic you just have to admit that you are dealing with two separate things and only inflict the complexity on the cases that actually need it.

So while it may make sense for Ecto.Changeset to offer structured error information that is so complex that you may need to use Ecto.Changeset.traverse_errors/2 to go through it all, that doesn’t automatically imply that a client of a context capability needs that same level of detail.

Even in the case of UI validation a simple “UI field”/“Field Error” key-value structure is probably enough - functional programming is all about data transformation - so transforming the error information to specifications of the context’s client before it passes through the boundary shouldn’t be an issue. And if it’s about function pipelines - what if stages produce similar looking errors but for some reason we need to know which particular stage produced the error. I guess we’ll have to write wrapper functions that add stage specific tags to the error tuple’s reason information.

So:

  • Contexts have boundaries.
  • Boundaries are supposed to (sometimes reverse but often) minimize dependencies. Dependencies are typically minimized via loose coupling.

Loose coupling leads to a specific type of relationship within and outside of the context boundary, with emphasis on reducing dependencies between the context boundary, the context implementation, and the context’s clients.

Loose coupling promotes the independent design and evolution of the context logic and implementation while still providing interoperability with its clients that have come to rely on the context capabilities. There are numerous types of coupling involved in the design of a context, each of which can impact the content and granularity of its boundary. Achieving the appropriate level of coupling requires that practical considerations be balanced against various context design preferences.

(Text liberally “repurposed and rephrased” from here).

I’m not exactly sure how this particular talk has managed to fly under the radar:

ElixirConf 2016 - Selling Food With Elixir by Chris Bell

Does this talk mention contexts? Not exactly. But for me it manages to check some of the boxes on the road towards contexts.

  • De-emphasizes the Rails way of doing things
  • De-emphasizes the database as the core of the application, relegating it somewhat to data backup and bootstrapping support.
  • Breaks the application down into distinct areas of domain concerns:
  • Store Availability
  • Order Scheduler
  • Order Tracker

Are those “areas of domain concerns” contexts? Hard to tell because there isn’t enough detail to judge the level of autonomy and decoupling that these “areas” have. And more to the point - for the time being the product has succeeded with the level of separation that it is currently implementing. Meanwhile some boundaries have already been defined - future development may discover that the boundaries have to be shifted or that they need to become more decoupled (fyi, the current version is running on a single node).

5 Likes