I it gives you a unique reference, you see it if you do :hackney.get('http://example.org/stream', [], <<>>, [:async]) in iex
yurko
0
peerreynders
My guess is that device pidfor IO.stream/2 or IO.binstream/2 is an IO server (process).
So I suspect that stream/binstream are responsible for sending {:io_request, from, reply_as, request} tuples to the specified IO server which is expected to serve {:io_reply, reply_as, reply} responses.
So the IO server would be entirely responsible for managing the network socket, including managing network errors and buffering any available payload that hasn’t been (io_)requested yet.
That being said there could be an easier way …
https://github.com/rvirding/tcp_io_server/blob/master/tcp_io_server.erl
Also:
HTTP Streaming in Elixir
Gun
2
dimitarvp
Thank you. While I moved on and this is no longer urgently relevant, it remained a point of interest which I will eventually pursue.
I like Go’s Reader and Writer way of doing things and I believe every language has to strive to emulate such concepts.
0
tfwright
Curious if you ever made any progress on this problem. I myself have found myself trying to implement a “source agnostic” streaming protocol and seem have to have run into exactly the same issues you have.
I am probably out of my depth here but it seems like a fundamental barrier might be that (most?) http servers operate on a “push” model (or maybe it is part of the protocol itself?) where the client is expected to listen for the next chunk whereas Elixir streams wait for the caller to “pull” the next part? So at minimum you are looking at implementing an intermediate process to handle the listening and then respond to the caller? But then I have to wonder how the GO API you mention works…
I’m curious about the theory here even if practically I end up just having the read the entire response into memory and then stream that 
0
dimitarvp
Yes but that’s not a problem; you can have a wrapper – the one that provides you a generic network stream abstraction – that simply reads ahead (or even reads it all) and then just serves you stuff one bite at a time.
Golang has interfaces (programming by contract) and you can get a Reader or Writer to practically everything and pass it to a function that utilizes either one while being 100% oblivious of the underlying implementation.
With Golang I have basically streamed a file uploaded to my app directly to S3 without ever having to store a single byte on the app server, for example.
The extremely useful rclone tool is written in Golang and uses and abuses the above idea all the time, to a crushing success.
As others have pointed out, something similar-ish can be achieved with Stream.resource; you can wrap stuff with it and provide it as an abstraction. Should be doable but I moved on long time ago and figured I’ll not try to piss against the wind. Truth is, most code does not need to be abstract and oblivious to the underlying implementation unless you’re a writing a library or a generic tool… and both are something that I do very rarely.
0
tfwright
…which is what I’m doing so I’ve self-assigned this particular problem
Yes but that’s not a problem; you can have a wrapper
By not a problem, you mean it’s technically possible? I’m trying to minimize deps in my dep so I’m trying to make this work with httpc. It has a stream option but it assumes the caller process will handle receiving each chunk, which is awkward at best. Probably will just read the entire response and stream that then and call it good enough for now
0
dimitarvp
engineeringdept
We have a process in our app that uses a GenStage producer to stream and parse an HTTP response in a single process. We use the Mint HTTP library in passive socket mode to receive chunks of binaries from the socket, passing them to the Saxy XML parser.
As the comment above says, you should be able to achieve something similar using Stream.resource. I’ve made an example that might help get you going: http_stream.exs · GitHub
3
dimitarvp
Hey, you know what? I’ll mark your answer as a solution, 5 years later. 
That code definitely gets the job done in this particular scenario and I’ve written a very similar one before.
The only thing I’d change is extract out the 3 functions passed to Stream.resource but it’s readable enough.
1
tfwright
That is indeed a very reasonable solution. Nice W for Mint providing that very neat passive mode to avoid having to handle incoming messages. Just took a brief look but it looks like it also has a stream API that might even simplify things further in many cases?
I’d really like to solve the problem in my app without the extra dep, but maybe I’ll find some clues digging through the Mint source.
0