Purpose of writing those files is to send them to Dropbox with a batch of 1000 as dropbox upload session supports 1000 images at a time.
Is there any better way to write images to disk? maybe in memory but I don’t know, any help would be wonderful also this operation is being done on Cuda GPU machine but I am not sure how I can use GPU for such purpose.
This process is user defined. user can ask for less/more than 1000 images and those can be one or multiple Task.async_stream’s…
I want to save them on disk so if the process broke or application gets a restart, I can resume the download from where it left.
Open the destination file in raw mode with a big buffer (512KB at least, if not a few megabytes even). This also avoids the file operations going through a single internal process – this happens to all non-raw-opened files.
Which basically gives you a much faster writing speed, utilising a generous 512KB buffer and 2 seconds of potential delay when the data will eventually make it to the disk (which should be plenty enough even on slow-ish servers).
From then on you can use IO.write or IO.binwrite on the returned handle. And don’t forget to call File.close at the end!
Also there are some ownership complications similar to sockets where only the process that opened the file can write to it, not sure if it’s a problem for the Task.async_stream approach described in OP (it shouldn’t be a problem if the file is opened and worked with within the same task process).
Since it sounds like you don’t need to persist the files in memory, you could mount a ramdisk so the files never have to touch the disk (assuming you have enough ram)