Getting Message and Errors from Broadway

Broadway catches exceptions and handles other errors by logging the error and “failing” the message. Is there a way to get a callback called to do something with that error as well? I would need the original message and the error (message, exception, etc).

My use case:

We’re collecting a bunch of data from a bunch of people over time. As the data gets collected it gets put on a queue and consumed by another app to coalesce that data into a bunch of “facts” (for lack of a better term) that we can query and report on. Any errors get sent back to the collection application by publishing a message to another queue with the original payload and the error reason. The sending application listens for these messages and collects them into a report for “quality control” to rectify.

Rabbit dead-letters don’t work because a nack doesn’t allow you change the message as far as I know. So we need to actually push another message. We built all of this ourselves before Broadway, but not quite as sophisticated as Broadway (e.g. termination process). I was looking to convert to Broadway and couldn’t see a way to do that. Any pointers?

Thanks.

2 Likes

Hi @geofflane!

We’ve been discussing the possibility of adding a callback (or something similar) for that purpose. For now, one thing you could try would be wrapping the code that might fail inside handle_message/3 or handle_batch/4 in a try/catch block, do your stuff and then re-raise the exception (or set the message as failed using Message.failed/2).

Not optimal, I confess. But it might do the trick until we have a built-in solution.

Cheers.

-marlus

1 Like

I’d love to see that kind of callback added. As you mention catching and rethrowing seems possible, just not an ideal solution. Thanks for the response!

In case anyone else stumbles across this in the future, since Broadway 0.5.0, the handle_failed/2 callback has been added to handle this: Broadway — Broadway v0.6.2