jschaeff
Hi fellows,
My application has to send data over HTTP to clients.
This data is retrieved from a list of files (containing basically time series) accessible via NFS.
One client request may mean touching hundreds of files.
The result has to be delivered in a consistent order so the data can not be mixed up.
Therefore, in order to mitigate the latency of accessing sequentially each file one after another, my idea is to copy locally in parallel all the needed files and then stream the data to the client sequentially (this will guarantee that timeseries get delivered in a predictable order).
Basically, I can:
- Copy files in parallel on a local work directory
- Then stream all local data with
Plug.Conn.chunk/2
But it is still not satisfactory. What I would like to do is:
- Parallely load NFS files from a designated offset until end offset into memory
- Stream the data in the order of the filenames
Do you have any hints to achieve this ?
Trending in Questions
Hello!
Suppose you are building workflow (order / task / payment) processing system with the following requirements:
Each workflow con...
New
Hey guys,
I’ve got a huge CSV ( around 10 GB ) that needs to be processed hourly
Do you guys have any suggestions what is the best prac...
New
Hello!
Could someone please give me a help/sample code, how to delete a file from s3 using waffle/waffle_ecto from Phoenix app.
I creat...
New
I have what I’ve heard referred to as a “lookup table” in my database. This is a way of assigning codes to common values. One common lo...
New
Hello,
I’m developing a online persistent chat system (what’s app) like using elixir/dynamodb/aws for a mobile app(flutter).
The diffic...
New
What approach to take when sending live updates to “random” users Hi! I have a question, I have a little chat app, and when I create a DM...
New
I think I’ve found a small improvement I could contribute to <%= web_namespace %>.CoreComponents (installer/templates/phx_web/compo...
New
Other Trending Topics
Hobbes is a low-level distributed database for the Elixir programming language.
Hobbes provides a simple, safe, and scalable storage lay...
New
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve.
They are GUI (Emerge) and State management (S...
New
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New
There are three potential reasons for members of this forum to have a look at https://vutuv.de
You are tired or annoyed of LinkedIn.
Yo...
New
Aludel - LLM Evaluation Workbench
Aludel is an embeddable Phoenix LiveView dashboard for evaluating and comparing LLM prompts across mult...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #ai
- #elixirconf-us
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming











Showing Posts 1 to 2- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
cmo
You gotta use the Erlang functions to skip to an offset.
jschaeff
Thanks for the hint. This seems to work as I expect:
I guess all I need to do after this is to send the data over Plug.