Streaming large files from server to server as the file is being written to

Questions / Help
1

The current situation

I have a Raspberry Pi that is connected to a IoT device running on Elixir. Right now I’m able to record the data from the device and save it to a file stored on the Raspberry Pi. I can then save that file from the raspberry to my computer, and then upload it to some server.

How I wish things should work

Now I’d like to skip the manual step.

I’d like to gather data from the IOT device, and as it is being recorded I’d like to stream the data to the server. It should still be stored to the Raspberry Pi, but it should also be uploaded to the server at the same time.

  • There should be guarantees there are no data losses
  • if connection between the Raspberry and the server is dropped, the upload should resume when the connection happens again
  • and similar, you get the idea I suppose

My question

Any tips on how to approach the problem would be much appreciated. Are there existing libraries I can rely on? Is it a tricky problem? Should I use something else instead of Elixir? Any other thoughts?

2

I am rather clueless on the topic but you can likely check out the Membrane sub-forum and the GitHub repo of the core library. People have done all sorts of embedded / IoT programs and media streams with Nerves and Membrane.