ijunaidfarooq

ijunaidfarooq

Writing 1K images to disk using Task.async_stream

I am downloading images through HTTP request with which I am getting a binary image, and writing it to a file such as

File.write(image_with_dir, image, [:binary]) |> File.close

this whole operation of getting HTTP request and then writing it to disk is done in

|> List.flatten()
|> Enum.sort()
|> Task.async_stream(&(inline_process.(&1, images_directory)), max_concurrency: System.schedulers_online() * 2, timeout: :infinity)
|> Stream.run

When decreasing max_concurrency the process got slow approx 2 minutes, also results of System.schedulers_online() is 8

but with current max_concurrency it faster but with this. Disk IO starts touching the limits

enter image description here

Purpose of writing those files is to send them to Dropbox with a batch of 1000 as dropbox upload session supports 1000 images at a time.

Is there any better way to write images to disk? maybe in memory but I don’t know, any help would be wonderful also this operation is being done on Cuda GPU machine but I am not sure how I can use GPU for such purpose.

This process is user defined. user can ask for less/more than 1000 images and those can be one or multiple Task.async_stream’s..

I want to save them on disk so if the process broke or application gets a restart, I can resume the download from where it left.

Most Liked

idi527

idi527

A file opened in raw mode wouldn’t work with IO.write and IO.binwrite, I think?

iex(1)> {:ok, fd} = File.open("some.file", [:raw, :write])
{:ok,
 {:file_descriptor, :prim_file,
  %{
    handle: #Reference<0.3732712904.755892237.167770>,
    owner: #PID<0.105.0>,
    r_ahead_size: 0,
    r_buffer: #Reference<0.3732712904.755892226.167656>
  }}}
iex(2)> IO.write(fd, "hello")
** (FunctionClauseError) no function clause matching in :io.request/2

    The following arguments were given to :io.request/2:

        # 1
        {:file_descriptor, :prim_file,
         %{
           handle: #Reference<0.3732712904.755892237.167770>,
           owner: #PID<0.105.0>,
           r_ahead_size: 0,
           r_buffer: #Reference<0.3732712904.755892226.167656>
         }}

        # 2
        {:put_chars, :unicode, "hello"}

    (stdlib 3.12.1) io.erl:565: :io.request/2
    (stdlib 3.12.1) io.erl:63: :io.o_request/3

:file.write can be used instead.

iex(2)> :file.write(fd, "hey")
:ok

Also there are some ownership complications similar to sockets where only the process that opened the file can write to it, not sure if it’s a problem for the Task.async_stream approach described in OP (it shouldn’t be a problem if the file is opened and worked with within the same task process).

axelson

axelson

Scenic Core Team

Since it sounds like you don’t need to persist the files in memory, you could mount a ramdisk so the files never have to touch the disk (assuming you have enough ram)

Where Next?

Popular in Questions Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New

Other popular topics Top

albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
marick
I had some trouble figuring out how to make many-to-many associations work. Once I got it working, I wrote a blog post. Because I’m a nov...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

We're in Beta

About us Mission Statement