TeaVent: Library for Event-Sourcing/The Elm Architecture (WIP)

Your Libraries & OS Mentoring Libraries
, ,
1

I am working on a new library called TeaVent(hex.pm.

It allows you to perform event-dispatching in a way that is a blend of Event-Sourcing and The Elm Architecture.

Rather than copy the documentation here verbatim, you can read it here, but as a quick Tl;Dr:

  • All events that come in are normalized, and then go through the reducer (event handler)-function configured by you. This allows events to modify the state of your system. The reducer is a pure function, which allows for easy reasoning and testing (including property-based tests).
  • Around this, you can add a context_provider, which, can transform your application state to look up that part that is significant for the current event. This allows you to use large systems that have databases, store something in GenServers, are distributed etc. to work with the system as well. (Instead of having a ‘single source of truth’ the ContextProvider sets up a ‘single source of truth for this event’) and after running the reducer, is able to use the results of the reducer-call to e.g. store it back in the database. Because this is separate from the reducer, it can be tested separately!.
  • Around this, you can add middleware to do instrumentation, or add things like database-transactions (doing these here mean that the ContextProvider can use a declarative db-interaction-style like e.g. Ecto.Multi, which again makes things easier to test!).
  • You can decide on different configurations for different environments, or a different one per-test, etc: Configuration is done in the final parameter passed to TeaVent.dispatch, or in the Application.env, falling back to some sensible defaults whenever possible, and raising errors on missing required options, or unknown options. This is based on the discussion on making better configurations in libraries.

The library strives to cater to each of the following needs:

  • ‘full’ Event Sourcing where events are always made and persisted, and the state-changes are done asynchroniously (and potentially there are multiple state-representations (i.e. event-handlers) working on the same queue of events).
  • a ‘classical’ relational-database setup where this can be treated as single-source-of-truth, and events/state-changes are only persisted when they are ‘valid’.
  • a TEA-style application setup, where our business-domain model representation changes in a pure way.
  • A distributed-database setup where it is impossible to view your data-model as a ‘single-source-of-truth’ (making database-constraints impossible to work with), but the logical contexts of most events do not overlap, which allows us to handle constraints inside the application logic that is set up in TEA-style.

What is missing from the current version is:

  • Better documentation for the current example.
  • A proper example (e.g. an example project) of how to use it with e.g. Ecto.
  • An even simpler ‘hello-world’ example that could be put in the documentation (/doctests).
  • Tests of all the bells and whistles.
  • Fixing all typespecs and making sure Dialyzer is happy with them.

And of course I am very eager to hear feedback from you!

11 Likes
2

I see there exactly the same problem as in almost any other ES library in Elixir: you only allow one instance of the application in whole project. Why not take example from Ecto.Repo and make it template method that will generate module for you and give you possibility to have multiple roots per project.

3

What would make you think that TeaVent would only allow one instance?

All configuration can be passed through the final argument to dispatch or dispatch_event, so you can have as many different set-ups as you wish. In fact, this is one of the main goals of this library, to allow you to have different set-ups in different parts of your test-suite!

1 Like
4

This looks really appealing. Will check it out.

1 Like
5

Aww, wish this existed 6 months ago, I basically wrote this in an Elixir bot that I wrote a long while back (although mine is introspectable and hot updateable via database mappings for web interface modification on-the-fly). ^.^;

EDIT: Just brought up my code, here are some ideas from it if you want to rip them out:

Dispatch an event:

EventPipe.inject(who, data)

The who is ran through a ProtocolEx protocol to translate it to an internal EventPipe structure, it accepts a variety of inputs from a Plug.Conn to grab the session data, Discord user, IRC user, system console, etc…

Once that conversion is complete then it ‘injects’ (dispatch is such a better name…) the data through the ‘pipeline’, which looks up in the database cache the MFA tuples, performs a set of checks, I have a matcher’ish-like syntax that is defined in the web browser interface for testing which should handle which type of thing, and all top-level matching event handlers of the auth/data combination get called with the data in the order of their priority (I always intended to auto-generate a ProtocolEx matcher module based on the database matchers but never got around to it).

Each of those matchers can then either handle the event auth/data as they wish, potentially (and often) re-injecting/dispatching more messages into the system (I don’t check for endless loops, it’s possible to happen, but as it’s only for my use I haven’t worried about such things as of yet), such as if someone sends a command or a webhook is called from github then it can dispatch a message to display (in a variety of rich and non-rich formats) to display in IRC, Discord, and websocket to a web-browser.

In addition its introspectable interface is like this:

# Add an event hook into the system, the database_name is a distinguisher so you can have multiple dispatching systems)
EventPipe.add_hook(database_name \\ :global, priority, matchers, extended_matchers, module, function, args)

# Delete a hook structure from the database, the structure you can get via `get_hooks` or so
EventPipe.delete_hook(database_name \\ :global, cb)

# Get all the hook structures from the database so you can display in an admin page, edit, modify, delete, whatever
EventPipe.get_hooks(database_name \\ :global)

And there is a variety of helper functions.

5 Likes
6

@OvermindDL1 Interesting! I can see the advantage of having separate dispatchers that you register as hooks to a ‘database’, because you make how an event is handled completely a concern of the consumer.

However, this does make the pipeline extremely implicit, (as well as the order being arbitrary for things with the same priority), which is why I chose not follow this approach by default in TeaVent.

Obviously, witing a TeaVent context manager or a TeaVent middleware that dispatches events in this way is very straightforward.

TeaVent is currently in use inside the development version of Planga (which will be merged upstream in approx. one/two weeks time as soon as it is finished, cleaned up and my colleagues are back in the office and the new implemented features are vetted), and I am very happy with how it has been able to improve the testing experience. Source here

But don’t take my word for it :smile:.

7

Not implicit, rather just defined in-database, which seems explicit to me. ^.^;

The order would be arbitrary for things with the same priority, however I cannot put something with the same priority into the database as it’s a unique column so it was not an issue I had to deal with.

I just needed something that is runtime configurable via database. :slight_smile: