I am creating a finite GenStage producer that reads the body of a hackney http request. Ideally, I’d like to hook this Producer up to a Flow using Flow.from_stage/1. I’ve been able to create a basic producer that reads the request, but don’t know how to shut down the flow after the body has been read. Is this possible?
A little more background
I’m trying to write a streaming file transfer application that gets a url (via hackney) and streams the response to an AWS bucket. I send off the http request and then read the request status and headers. If I get a good status code, I start streaming the body to AWS via Flow. Otherwise, I close the connection and return immediately.
I’ve tried using Stream.resource/3 to create a stream that reads the body of the response, and pass that to the Flow using Flow.from_enumerable/1. However, it turns out Flow.from_enumerable/1 reads the stream in a new process, and you can’t read a hackney response from a process that doesn’t own the connection.
So my thought was to create my own Producer that I can transfer ownership of the hackney connection to, and have it send the results into the flow using Flow.from_stage/1. However, when I reach the end of the body, how do I shutdown the entire flow like is done when you use Flow.from_enumerable/1 on a finite collection?
