How to acknowledge a Broadway message asynchronously?

Questions / Help
, , ,
1

Hi,

Scenario

I have a SQS queue with messages, each message is targeted to a specific user, each user joins his own Phoenix channel via websocket. I’m using Broadway to keep pulling the messages. In it’s handle_message/3 I use Phoenix’s broadcast/3 to push the message to the user:

Endpoint.broadcast("user:#{user_id}", "message", message)

I would like to make sure the user has received the message before I acknowledge it to Broadway/SQS.

Since the broadcast/3 doesn’t return the delivery result, I’m planning to make my JavaScript library to push an “ACK” back to the channel. I would include Broadway’s message_id in both messages.

The problem

How can I acknowledge the message to Broadway from the Phoenix’s channel handle_in/3?

As a side note, I started wondering if Broadway is the right tool for my use case, given the handle_message/3 documentation says:

Basically, any CPU bounded task that runs against a single message should be processed here.

but my task is more an I/O thing.

Any help is much appreciated.

2

Yeah you probably want to do the broadcast in handle_batch because it would let you push a batch of messages from SQS to the end user, and then do a more efficient ACK back to SQS.

As far as getting the ACK from the client side, I think your only option is to block handle_batch until you get a message back from the client. Getting this ack back might be a bit tricky, perhaps you’ll have to have the broadway producer subscribe to an ack topic before pushing the messages out?

3

Yeah, I have no idea.

I was hoping Broadway could allow us to configure it to do not call handle_batch/4 automatically. Then I would call it myself once the last ACK arrives or times out.