I’m wondering if I could get some insight on how people are doing the following.
The flow is simple, and I’m sure there are a lot of people already doing it, but I need to expose an endpoint, let’s say
POST events (authentication/authorization will also happen in here) where other services can send events , those events should be persisted into kafka, and then consumed by elixir/broadway to do some data manipulation and other stuff.
The part that I’m curious about is how to send those events to kafka. I was hoping to use Broadway to manage it too, and make use of all the features it provides, but to do this I would have to implement some kind of intermedia storage, to put those http events, and being able to read them using a custom producer (I couldn’t find any other way).
I’m trying to simplify the complexity and underlying technology between the http event and kafka, but without losing control and “supervision” of those events.
Appreciate you all for any kind of input you could provide!
Thanks a lot!