I think I have Broadway setup pretty well. Here is a demo video where I am taking in log entries via API request > GenServer > reading and inserting them into Big Query with Broadway > then I’m able to use DataStudio to explore those (via a Google Cloud API integration).
Only thing I’m not sure about is how to ack unsuccessful messages. Right now I’m buffering things into this GenServer. As I read them from the buffer I move them to the read_receipts queue. When I ack them I just pop them from that queue and don’t do anything with it. That needs to ultimately be a different data structure so I can pick out specific messages but curious as to where I handle the unsuccessful Big Query write attempts?
Here is the producer: https://github.com/Logflare/logflare/blob/master/lib/logflare/broadway_buffer/buffer_producer.ex
And the pipeline: https://github.com/Logflare/logflare/blob/master/lib/logflare/table_big_query_pipeline.ex
Broadway is pretty rad btw! Appreciate it!
