jschaeff

jschaeff

Hi fellows,

My application has to send data over HTTP to clients.
This data is retrieved from a list of files (containing basically time series) accessible via NFS.
One client request may mean touching hundreds of files.
The result has to be delivered in a consistent order so the data can not be mixed up.
Therefore, in order to mitigate the latency of accessing sequentially each file one after another, my idea is to copy locally in parallel all the needed files and then stream the data to the client sequentially (this will guarantee that timeseries get delivered in a predictable order).

Basically, I can:

  • Copy files in parallel on a local work directory
  • Then stream all local data with Plug.Conn.chunk/2

But it is still not satisfactory. What I would like to do is:

  • Parallely load NFS files from a designated offset until end offset into memory
  • Stream the data in the order of the filenames

Do you have any hints to achieve this ?

Showing Posts 1 to 2

cmo

cmo

You gotta use the Erlang functions to skip to an offset.

{:ok, f} = :file.open(file, [:binary, :read])
{:ok, ^offset} = :file.position(f, offset)
jschaeff

jschaeff OP

Thanks for the hint. This seems to work as I expect:

{:ok, f} = :file.open("FR.CIEL.00.HHZ.D.2023.001", [:binary, :read])
{:ok, data} = :file.pread(f, 8192, 4096)

I guess all I need to do after this is to send the data over Plug.

— All posts loaded —

Where Next? Top

Trending in Questions Top

stjefim
Hello! Suppose you are building workflow (order / task / payment) processing system with the following requirements: Each workflow con...
New
Blokh
Hey guys, I’ve got a huge CSV ( around 10 GB ) that needs to be processed hourly Do you guys have any suggestions what is the best prac...
New
kszambelanczyk
Hello! Could someone please give me a help/sample code, how to delete a file from s3 using waffle/waffle_ecto from Phoenix app. I creat...
New
Onor.io
I have what I’ve heard referred to as a “lookup table” in my database. This is a way of assigning codes to common values. One common lo...
New
jaybe78
Hello, I’m developing a online persistent chat system (what’s app) like using elixir/dynamodb/aws for a mobile app(flutter). The diffic...
New
Trolleger
What approach to take when sending live updates to “random” users Hi! I have a question, I have a little chat app, and when I create a DM...
New
widianto
I think I’ve found a small improvement I could contribute to <%= web_namespace %>.CoreComponents (installer/templates/phx_web/compo...
New

Other Trending Topics Top

garrison
Hobbes is a low-level distributed database for the Elixir programming language. Hobbes provides a simple, safe, and scalable storage lay...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New
netoum
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New
wintermeyer
There are three potential reasons for members of this forum to have a look at https://vutuv.de You are tired or annoyed of LinkedIn. Yo...
New
webofbits
Aludel - LLM Evaluation Workbench Aludel is an embeddable Phoenix LiveView dashboard for evaluating and comparing LLM prompts across mult...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews