miguelszerman

miguelszerman

Eager vs Lazy Loading? (Elixir getting started guide)

In the Enumerable chapter of the Elixir getting started guide there is a section called Lazy vs Eager. However, it never explains what Lazy or Eager means in the Elixir context.

There is an external mini-explanation on educative.io:

Enum, being eager, produces a whole list of numbers after each operation in the script until the result is reached. Conversely, Stream, being lazy, creates a stream that represents a function without executing it straight away.

A better explanation with the trade-offs, common use cases, and links to further resources would be nice to include in the guide.

Marked As Solved

josevalim

josevalim

Creator of Elixir

Apologies for the confusion, your description was great. However, I was eyeing @miguelszerman’s summary, because it is small and therefore a perfect fit for an introductory guide. :slight_smile: Good news is that it is less work on your plate!

Also Liked

dimitarvp

dimitarvp

I don’t know about links and I am not a computer scientist by education but as a seasoned programmer I can offer you the following explanations.

— Eager loading —

Everything gets loaded in memory right away. Example 1:

"~/Downloads/countries.json"
|> File.read!()
|> CSV.parse_string()
|> do_stuff_with_records()

File.read! loads the entire file into a single binary (string) and then passes it to CSV.parse_string which also works with an entire binary. That’s eager loading – everything is there to be used in one go.

Example 2:

list_of_1000_elements
|> Enum.map(& &1 * 3)
|> Enum.map(& &1 / 7)
|> Enum.filter(& &1 / 2 == 0)

Here you don’t just work with one list of 1000 elements; you work with four of them in total since each Enum.map or Enum.filter produces a new list that’s also loaded entirely in memory. Meaning the memory for all four lists will be allocated and used until they are thrown away.

Something like 95% of the time you don’t care and it’s fine. But every now and then this is an awful idea because you don’t know beforehand how many records do you have to process in advance. Which brings us to…

— Lazy loading —

Stuff that you work with gets loaded in memory in chunks / batches. You never load the entire thing in memory.

Let’s take the Example 1 from above and turn it into lazy-loading code.

"~/Downloads/countries.json"
|> File.stream!()
|> CSV.parse_stream()
|> do_stuff_with_records()

Notice how we replaced File.read! with File.stream! and CSV.parse_string with CSV.parse_stream. You should read quickly on these functions but basically they operate with an Enumerable that allows them to pull data on demand (in batches). OK, maybe not the best example because you have to defer to documentation for an external library so let’s go to Example 2:

list_of_1000_elements
|> Stream.map(& &1 * 3)
|> Stream.map(& &1 / 7)
|> Stream.filter(& &1 / 2 == 0)
|> Enum.to_list()

The Stream functions are usually identical with those with the same names from Enum and do the same thing, only they never operate with the entire list given. In this case you only work with two lists in total: the original one and the resulting one which is produced by feeding a stream to Enum._to_list (NOTE: you can merge the last two steps by just doing Enum.filter(& &1 / 2 == 0) and it will have the same effect, but I opted for slightly longer code for illustrative purposes).

The very good thing about this approach is that the original list doesn’t even have to be loaded into memory as well. Example 3 and that one is much closer to real-life scenarios:

{:ok, list_of_results} =
  Repo.transaction(
    fn ->
      an_ecto_query
      |> Repo.stream(max_rows: 1000)
      |> Stream.map(...)
      |> Stream.filter(...)
      |> Stream.flat_map(...)
      # etc. processing steps for each record
      |> Enum.to_list()
    end,
    timeout: :infinity
  )

I and many others have successfully used code like the above to process dozens of millions of DB records, while the code never loads more than 1000 at the same time.


Now this is not super formal or strictly adhering to the scientific definitions, surely, but is more like an answer to the question: “What does eager / lazy loading means when programming [in Elixir]?”.

TL;DR – it’s usually a protection from bursty memory loads. And it can sometimes slow down a competing Enum implementation if you go too micro (on my machines I never use Stream unless I have to operate with more than 3000-4000 records at a time).

dimitarvp

dimitarvp

Hahaha. I got greatness-blocked! :003:

dimitarvp

dimitarvp

Swamped with work and this post was a bit of an anxious procrastination, admittedly. :icon_redface:

I promise I’ll find a time slot in the next several days and will PR this – do you mind references to external libraries, or you are OK with them?

Last Post!

dimitarvp

dimitarvp

Hahaha. I got greatness-blocked! :003:

Where Next?

Popular in Questions Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 130286 1222
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 54006 488
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New

We're in Beta

About us Mission Statement