almir-neto

almir-neto

There is some advantage to use `File.open` instead `File.write`?

I know that File.write performs two operations: it opens the file and then writes to it. However, considering that File.open followed by IO.write avoids one operation for each iteration, I want to know: when I’m building a lazy pipeline, is it recommended to use File.write for each iteration, or should I keep the file open and just write to it? Is there any significant impact between these two approaches?

Could someone clarify the pros and cons of these two approaches for me?

PS: I am focused on impacts for lazy pipelines like this:
Stream.map(…)
|> Stream.reduce(…)
|> Stream.with_index(…)
|> Stream.map(write_on_file)

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

In these sorts of patterns I would also checkout Stream.into(File.stream!(path))

dimitarvp

dimitarvp

I’ve used File.open and IO.write / IO.binwrite with a Stream like you, dozens of times, and never benchmarked it against having File.write at each step. I simply assumed the latter is a terrible idea and can never be faster. Main reason might not even be speed per se; it’s just that opening and closing a file can be risky under load and in more constrained machines (which, let’s be real, are most cloud offerings out there, even the mainstream ones; they regularly put your apps and databases on I/O-throttled machines) – it might actually fail with “out of memory” or “not enough file handles” etc., I’ve seen those even on machines with 64GB of RAM and 2TB SSD storage.

TL;DR do as little syscalls as at all possible. We can’t optimize adequately for that when we work with a higher-level language with Elixir but at least we can use the little tools we have, like “don’t open and close file 500 times in a loop”.

…And all of that changes if each step to write to the file is seconds or minutes apart, of course (i.e. you stream data from a slower remote node).

lud

lud

I’m not sure I understand the question. If you need to do multiple writes to a file in a loop then yes, File.open is made for it. If you have the complete content of the file to write already, then use File.write only once.

Where Next?

Popular in Questions Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
Kurisu
For example for a current url like http://localhost:4000/cosmetic/products?_utf8=✓&query=perfume&page=2, I would like to get: ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> somethi...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New

Other popular topics Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New

We're in Beta

About us Mission Statement