Design of an Elixir based rules system with multiple rules-bases

Questions / Help
1

I’m looking for guidance on the design concept for an application we are building using Elixir/Ecto/Phoenix* which uses a rule-based system that observes the behaviour of a group of users and interacts with them.

Our application will have a number of groups (think management team or board members) and will build a facts-base from the data provided by a group and its activities. A set of rules will then be used to drive some application behaviours based on the data.

A rules-engine approach seems very fit to this kind of activity (taking into account the warnings of people like Martin Fowler about the complexity of such systems).

In thinking about how to design it I am gravitating towards the idea that we should use a GenServer with the facts database of each group as it’s state. That is rather than having one big monolithic GenServer as a rules processor for the entire DB, we create a separate GenServer instance for each group.

I see a number of advantages:

  1. Each GenServer will be smaller, having only the data for one group to turn into its facts base.

  2. Each GenServers could be distributed to reduce individual host load (esp. if we used something like fly.io for the app)

  3. We can use Phoenix pub/sub to glue everything together so that the app and rules engine can communicate

  4. It keeps us all one stack BEAM/Elixir/… so reduces the required knowledge surface area of the team

Three issues are on my mind:

  1. Building a rules engine

Building a “naive” rules engine isn’t especially difficult. But is is more code to implement. There are not, so far as I can see, any released native BEAM rules engines although projects like Bob Sollish’s Rules_Engine and the Retex library are potentially useful starting points.

Of course we could use something like DROOLS or JESS which are big established players. I’m not keen (a) because I would like to stay on the BEAM platform and not introduce another and (b) because they are designed for big complex organisations with big complex rules and it seems to me that big complex rules is a rules-base smell.

This latter part is quite important as big complex rules systems employing chaining can be hard to reason about. I’d almost take the “naive” point as a useful constraint.

  1. How to orchestrate lots of GenServers

I am not sure how to orchestrate all these various GenServers. For example if we have hundreds of groups, what is reponsible for saying “Oh, there’s a new group here that needs a rules-engine, go spin up a GenServer for them”. If the app is restarted, what makes sure everything comes up properly?

  1. Preservation of state

How to preserve the state of the GenServer fact-base itself. The facts will, ultimately, come from the application database through a process of translation. But this could be unwieldy for startup times as the fact-base gets bigger. Maybe we don’t need to consider this now but it could be an issue long-term.

This is my first foray into the design of a (albeit modestly) complex Elixir application and I would be grateful for any thoughts in response to what I’ve written, suggestions for people/things to read or think about, or suggestions more directly about what we are attempting.

Many thanks.

Matt

2 Likes
2
  1. Building a rules engine

There are some rule engine projects in this area in development, Retex is the furthest I know of. I’m working on a similar effort (not at all ready for use yet), but is more targeted at “workflows” rather than rule evaluation so does compile and evaluate rules, but also DAG pipelines and accumulations.

I’d look at the naive approach. A simple approach is to take boolean conditionals as the first steps from the root of a directed acyclic graph (DAG) :root → (many) conditionals → steps/reactions. Evaluation would be brute force looping through every conditional / rule left hand side for each assessment but can be powerful in simple cases.

GitHub - ympons/expreso: ☕ A boolean expression parser and evaluator in Elixir. is worth a look for user defined rules.

  1. How to orchestrate lots of GenServers

For something like a chat system where a set of rules might be composed for a given chat room a Dynamically Supervised and Registered set of GenServers seems like a good place to look.

This generally consists of a Registry (for finding a pid from an external id), a Dynamic Supervisor (for supervising runtime spawned processes like a new chat room).

  1. Preservation of state

It depends. If your rules are elixir / erlang you can use :erlang.term_to_binary and :erlang.binary_to_term with some of the fun and dangerous (if user inputs are allowed) functions in the Code module. At that point rule storage is just storing binaries in Postgres via Ecto or similar. If you need stateful rule context you will need to store some kind of fact base, but if it’s stateless just the rule set itself needs to be known in context of a chat room.

Hope this helps.

2 Likes
3

I am familiar with Retex but not with expreso while I will take a look at, thank you. I am also familiar with Bob Sollish’s Rules_Engine which is a port of Java’s EasyRules. This is a naive rules engine implementation, unfortunately, he didn’t put it under an open-source license (I have asked him if he would go back and do this).

I’m actually seeing the need for RETE on modern hardware as a potential sign of problems. Rules-based systems are a powerful construct but I think the constraint of having to stay within a naive implementation could help avoid complexity.

This isn’t a chat application, more in the line of business intelligence. We have groups doing things and it seems like it would be a good idea to operate a separate rules-system for each group.

I guess what I am missing is how a component knows which rules engines to start and (in a possibly distributed environment) where.

Interesting, I’ll have a think about this approach.

Keen to hear more about your own efforts.

Thanks.

Matt

4

What about using or porting SERESYE?

Isn’t Erlang/Elixir stack with its recursive matching nature itself is an off-the-shelf rule engine?

What I feel that some data must be specially prepared, and then all rules can be just implemented as a native Erlang functions.

1 Like
5

You can have another GenServer, a manager responsible for listening for Group create/update/delete.

I don’t think it’s that difficult to know when a group is added, or removed, or updated… You could use postgresql notifications, or publish an event when in the context module.

Once the data is in the database, You could use the manager to query the db on start, and then load data to start dynamic supervisors, in the handle_continue of the manager.

UPDATE: Sorry @MrDoops, reply is for @mattmower, clicked on the wrong reply button :slight_smile: