Recently I ran into a problem similar to the one mentioned in Is there a way to split a stream into two outputs? similar to `tee` on linux. Essentially, I wanted to split a stream into two streams. After some googling around to figure out how other languages solve this, I came across this paper http://benl.ouroborus.net/papers/2016-polarized/dpdf-FHPC2016-preprint.pdf. This kind of cleared up things for me. Please read the paper up to section 2.4 before reading further. I have also written a blog post about how this applies to elixir http://ananthakumaran.in/2017/11/28/stream.html.
I want to start a discussion about the lack of transform operations for
Collectable protocol in the standard library. Any transform operation falls into one of the three categories
only supports source -> Eg zip, concat etc
only supports sink -> Eg into
supports both source & sink -> Eg map, filter
Because the standard library doesn’t provide any transforms for
Collectable, most of the libraries just support the source type streams.
Eg. https://github.com/ne-sachirou/stream_gzip, with proper api, this could be implemented in such a way that it accepts both source & sink type streams.
Eg. https://github.com/ex-aws/ex_aws_s3 this library allows a stream to be upload to s3. Again, it expects source type stream.
I guess by not providing all the possible transform operations, we are reducing the flexibility of the users and make it much more difficult to implement any kind of stream that needs to be forked.