Stream Source and Sink

Recently I ran into a problem similar to the one mentioned in Is there a way to split a stream into two outputs? similar to `tee` on linux. Essentially, I wanted to split a stream into two streams. After some googling around to figure out how other languages solve this, I came across this paper http://benl.ouroborus.net/papers/2016-polarized/dpdf-FHPC2016-preprint.pdf. This kind of cleared up things for me. Please read the paper up to section 2.4 before reading further. I have also written a blog post about how this applies to elixir http://ananthakumaran.in/2017/11/28/stream.html.

I want to start a discussion about the lack of transform operations for Collectable protocol in the standard library. Any transform operation falls into one of the three categories

only supports source -> Eg zip, concat etc
only supports sink -> Eg into
supports both source & sink -> Eg map, filter

Because the standard library doesn’t provide any transforms for Collectable, most of the libraries just support the source type streams.

Eg. https://github.com/ne-sachirou/stream_gzip, with proper api, this could be implemented in such a way that it accepts both source & sink type streams.
Eg. https://github.com/ex-aws/ex_aws_s3 this library allows a stream to be upload to s3. Again, it expects source type stream.

I guess by not providing all the possible transform operations, we are reducing the flexibility of the users and make it much more difficult to implement any kind of stream that needs to be forked.

:thumbsup: for a contramap operation on Collectable.

I’m not sure if generalized stream splitting is possible given Elixirs reducee design?
The stream can only be reduced once.

I think GenStage and Flow might be the general solution to this problem.

1 Like

Could this be relevant? (at 37m)

1 Like