I think I have Broadway setup pretty well. Here is a demo video where I am taking in log entries via API request > GenServer > reading and inserting them into Big Query with Broadway > then I’m able to use DataStudio to explore those (via a Google Cloud API integration).
Only thing I’m not sure about is how to
ack unsuccessful messages. Right now I’m buffering things into this GenServer. As I read them from the buffer I move them to the
read_receipts queue. When I
ack them I just pop them from that queue and don’t do anything with it. That needs to ultimately be a different data structure so I can pick out specific messages but curious as to where I handle the unsuccessful Big Query write attempts?
Broadway is pretty rad btw! Appreciate it!