Hi all,
I’m tinkering around the idea of streaming data through an external program (think streaming video through ffmpeg
command and receiving the output back) from the last few weeks. Mainly focused on communicating with long-running programs with back-pressure. After exploring many approaches I settled on this. ExCmd uses named FIFO to solve back-pressure and other issues. It also uses odu (which is based on goon) to fill gaps in the erlang ports.
Currently, it’s at an early stage. I’m still thinking about the interface it should provide to expose all its functionality for different use cases effectively.
Please check it out and share your feedback
Background
Why not use built-in ports?
- Unlike beam ports, ExCmd puts back pressure on the external program
- Proper program termination. No more zombie process
- Ability to close stdin and wait for output (with ports one can not selectively close stdin)
While exploring the options, I also played around another approach, which does not use named FIFO. Its more like GenStage, the receiver beam process “demands” external program for output using stdin and stdout., but it has its own set of other issues.