How to write to a file that combined by multiple blocks in shorter time

Questions / Help
, ,
1

I want to write a big file, but the file is from 1000 blocks, also the blocks are from network. so I think better not doing that in a sequential manner like:

  • first read all blocks or wait until block ready to write like 0,1,2,3,4,…wait 5 to come

I want to do like this:

  • preallocate 1000Mb file
  • spawn 1000 processes to get the blocks
  • if block 51 ready then write 51 to 51*1MB offset
  • do this until write all.

Question:

  • does Elixir support pre allocate?
  • can I write to same file at diff offset simultaneously without locking issue?
2

Are the blocks arriving out of order, e.g. can you get blocks 1, 2, 5, 6, 7, 3, 4 etc.?

3

Yes, arriving out of order.

4

I think you should be able to do random access writes with :file.pwrite/3. If you open the file in :raw mode then you also write directly the file, not via the IO process (if I’ve understood this correctly).

1 Like
5

How to open a file with random access? or just pwrite will do?

6

Thanks pwrite works, I have 800 blocks file write to disk dropped from 40 sec seq write to 5 sec in random access write.