Broadway Demo and Question

I think I have Broadway setup pretty well. Here is a demo video where I am taking in log entries via API request > GenServer > reading and inserting them into Big Query with Broadway > then I’m able to use DataStudio to explore those (via a Google Cloud API integration).

Only thing I’m not sure about is how to ack unsuccessful messages. Right now I’m buffering things into this GenServer. As I read them from the buffer I move them to the read_receipts queue. When I ack them I just pop them from that queue and don’t do anything with it. That needs to ultimately be a different data structure so I can pick out specific messages but curious as to where I handle the unsuccessful Big Query write attempts?

Here is the producer:

And the pipeline:

Broadway is pretty rad btw! Appreciate it!