Message Handler Pattern

So to set the scene, i am new to Elixir and this might be something i can easily search for but i could not find anything on the topic.

In Python i use a pattern of decorators and register them as handlers and then run a consumer for some message queue and route the messages to the specific handlers, it looks something like this:

    app = Consumer()
    def handle_message(topic, data):
        # do something with the data

Inside the Consumer class a dict is built out that looks something like this:

    handlers = {
        "my.topic": [function<>, function<>]

When a message is received it looks for a matching topic and executes all the functions that are subscribed to this topic. It is similar to a really simple URL router for a web framework Is this type of pattern possible in Elixir? Is there a better way that fits into the Elixir paradigm?

I am open to suggestions, its not a work thing, i am learning Elixir in my own time still and trying to implement solutions that i have written in Python and JS in Elixir.

I am not aware in such a pattern in Elixir. Here, pubsub generally involves processes, so if you want a common handler for multiple topics you would start a process, then subscribe to the topics, and then you process would be sent messages that you can handle.

Otherwise multiple processes could subscribe to a specific topic of their choice.

You always have to define explicitely where (in which process) the messages are handled, because it is a process that is subscribed, not a function.

That being said it could be easy to write a library that would manage its own processes and just declare module/function as handlers, but I am not sure wether it would be useful.

It‘s also not really clear if this is supposed to involve processes in the first place. E.g. telemetry allows for registering listeners to events, where handlers are executed inline.

1 Like

I belive your question has an XY problem.
I think the best approach is to describe your problem without abstractions over it for example:
“i want to handle ws messages in elixir”.
“i want to consume a message queue in elixir”
“I want to build a pubsub system in elixir”

When you com with a solution such as “message handler pattern” and look for alternatives, is difficult to understand because we don’t know what you’re trying to solve. That’s even more difficult to help you, if your solution comes from a OO pattern to a functional paradigm solution.

For us to help you better, can you explain what is your problem first?

So all fair comments so far, i will explain more about what i am trying to do.

I am experimenting with Fluvio a new message queue similar to Kafka, I have tested and played with their API clients for Rust, JS and Python (Python being my day job). I am keen on learning more Elixir (been drawn to it for a while) and wanted to see how easy it was to connect to a message queue and stream the data. A common task from my day job, I work in IoT and building data pipelines.

My end goal would be to use the Fluvio Rust API to create a NIF that will consume the data and route them to another function within a module.

I have came across Broadway and that means I could add a custom transport option for Broadway, but I am unsure of how to combine the Rust API in this. Another idea I thought of is could I use a supervised process (Is that a GenServer?) per topic? The only issue i can see here is then i might be making lot’s of connections to the broker.

Hopefully that helps explain the context


There are several layers on your question, I’ll try to address them without going too deep.

  • On rust interactions you’re right to consider Rustler, on how to model that into your system, it really depends on how light/expensive is to run that layer with your broker.
  • On using broadway, most of the cases related to ingesting data from message brokers and similar stuff, broadway would fit perfectly, there are some edge cases that it doesn’t, but it’s more related on how you’re modeling to use those systems then with broadway itself.

On your use case:

If your idea is to broadcast data between systems, i’d try to play in Elixir with something that already have an sdk/proper client for it.
I’d suggest playing with kafka or rabbitmq using broadway to consume it. Building abstractions to interact with a message broker is a very complex task, would make you learn the internals of the broker to properly abstract the client side of it. So if you’re just learning and experimenting, try something that similar properties to fluvio and start from there.