So to set the scene, i am new to Elixir and this might be something i can easily search for but i could not find anything on the topic.
In Python i use a pattern of decorators and register them as handlers and then run a consumer for some message queue and route the messages to the specific handlers, it looks something like this:
app = Consumer()
@app.subscribe("my.topic")
def handle_message(topic, data):
# do something with the data
Inside the Consumer class a dict is built out that looks something like this:
When a message is received it looks for a matching topic and executes all the functions that are subscribed to this topic. It is similar to a really simple URL router for a web framework Is this type of pattern possible in Elixir? Is there a better way that fits into the Elixir paradigm?
I am open to suggestions, its not a work thing, i am learning Elixir in my own time still and trying to implement solutions that i have written in Python and JS in Elixir.
I am not aware in such a pattern in Elixir. Here, pubsub generally involves processes, so if you want a common handler for multiple topics you would start a process, then subscribe to the topics, and then you process would be sent messages that you can handle.
Otherwise multiple processes could subscribe to a specific topic of their choice.
You always have to define explicitely where (in which process) the messages are handled, because it is a process that is subscribed, not a function.
That being said it could be easy to write a library that would manage its own processes and just declare module/function as handlers, but I am not sure wether it would be useful.
Itâs also not really clear if this is supposed to involve processes in the first place. E.g. telemetry allows for registering listeners to events, where handlers are executed inline.
I belive your question has an XY problem.
I think the best approach is to describe your problem without abstractions over it for example:
âi want to handle ws messages in elixirâ.
âi want to consume a message queue in elixirâ
âI want to build a pubsub system in elixirâ
When you com with a solution such as âmessage handler patternâ and look for alternatives, is difficult to understand because we donât know what youâre trying to solve. Thatâs even more difficult to help you, if your solution comes from a OO pattern to a functional paradigm solution.
For us to help you better, can you explain what is your problem first?
So all fair comments so far, i will explain more about what i am trying to do.
I am experimenting with Fluvio a new message queue similar to Kafka, I have tested and played with their API clients for Rust, JS and Python (Python being my day job). I am keen on learning more Elixir (been drawn to it for a while) and wanted to see how easy it was to connect to a message queue and stream the data. A common task from my day job, I work in IoT and building data pipelines.
My end goal would be to use the Fluvio Rust API to create a NIF that will consume the data and route them to another function within a module.
I have came across Broadway and that means I could add a custom transport option for Broadway, but I am unsure of how to combine the Rust API in this. Another idea I thought of is could I use a supervised process (Is that a GenServer?) per topic? The only issue i can see here is then i might be making lotâs of connections to the broker.
There are several layers on your question, Iâll try to address them without going too deep.
On rust interactions youâre right to consider Rustler, on how to model that into your system, it really depends on how light/expensive is to run that layer with your broker.
On using broadway, most of the cases related to ingesting data from message brokers and similar stuff, broadway would fit perfectly, there are some edge cases that it doesnât, but itâs more related on how youâre modeling to use those systems then with broadway itself.
On your use case:
If your idea is to broadcast data between systems, iâd try to play in Elixir with something that already have an sdk/proper client for it.
Iâd suggest playing with kafka or rabbitmq using broadway to consume it. Building abstractions to interact with a message broker is a very complex task, would make you learn the internals of the broker to properly abstract the client side of it. So if youâre just learning and experimenting, try something that similar properties to fluvio and start from there.
