coen.bakker

Is this `seeds.exs` idiomatic Elixir?

One of the requirements of the Book Club exercise on elixirland.dev is to seed data.

I am referring to this requirement in the exercise.

## Requirements

...

  ### **Seeding**
  * Running mix ecto.setup creates the database tables but also seeds the database
  * Seeding inserts 4,000 books that each have 10 pages
  * Some seeded books have an active page, but not all
  * Seeding is fast

How happy or unhappy would you be if a co-worker would create a pull request containing this seeds.exs? Is it written in idiomatic Elixir? Use of comments, etc.

import Ecto.Query, only: [from: 2]
alias BookClub.Repo
alias BookClub.Books.{Book, Page}

# Setup
# Sets log level to :info for performance
initial_log_level = Logger.level()
Logger.configure(level: :info)
IO.puts("Start database seeding")
start_time = System.os_time(:millisecond)

# Constants
n_books = 4000
n_pages_per_book = 10
inserted_at = NaiveDateTime.utc_now(:second)
batch_size = 800

# Insert 4,000 books
books =
  for _ <- 1..n_books do
    %{
      title: XlFaker.generate_title(),
      inserted_at: inserted_at,
      updated_at: inserted_at
    }
  end

books
|> Enum.chunk_every(batch_size)
|> Enum.each(&Repo.insert_all(Book, &1))

# Batch insert 10 pages per book
# Gives some books an active page
book_ids =
  from(b in Book, select: b.id)
  |> Repo.all()

pages =
  book_ids
  |> Enum.flat_map(fn book_id ->
    pages =
      for i <- 1..n_pages_per_book do
        %{
          book_id: book_id,
          content: XlFaker.generate_page(),
          number: i,
          status: :inactive,
          inserted_at: inserted_at,
          updated_at: inserted_at
        }
      end

    List.update_at(
      pages,
      :rand.uniform(n_pages_per_book) - 1,
      &Map.put(&1, :status, Enum.random([:active, :inactive]))
    )
  end)

pages
|> Enum.chunk_every(batch_size)
|> Enum.each(&Repo.insert_all(Page, &1))

# Teardown
end_time = System.os_time(:millisecond)
IO.puts("Finish database seeding")
IO.puts("Seeded #{n_books} books in #{end_time - start_time}ms")
Logger.configure(level: initial_log_level)

22 comments

/phoenix #idiomatic-elixir #learning-elixir #seed #elixirland

0 854 22

2024-08-30 19:18:06 UTC

First Post!

LostKobrakai

I don’t think the log level should be overwritten. Either the system is configured to info level and info is logged or it isn’t. I’d also consider changing the IO.puts to use logger.

The book ids could be returned from the Repo.insert_all calls with the :returning option + a flat_map.

And as with all code generating data: This “works”, but it doesn’t really do much in terms of making sure it continues to work over time and in the face of changes to the system it seeds. There’s a reason tests commonly use factories instead of code like this. Though this is one piece of code, tests are usually quite many, so it’s less of a problem.

Post #1

Most Liked

fuelen

I’d be okay if this script is just a temporary solution. The code itself is good enough. My concerns are not about style of Elixir code but more about approach in general. It works for simple cases, but as the project grows, I’d not put

* Seeding is fast

to the requirements.
What is more important for me is correctness of data. The approach of inserting raw data will quickly become a mess on several dozen tables. I’m a proponent of using business functions in seed scripts. I’m okay to wait a bit longer. Inserting seed data can be sped up by parallelization (Task.async_stream or flow can help). Because of that, having progress bars in seeds is also a good thing (owl can help).

The number of inserted data must be configurable, so it is possible to specify tiny numbers to run seeds as a part of the test suit. Just create 1 record for each record type to ensure that the script doesn’t fail. Most likely, you won’t be happy when the script stops working when you need it the most

On one of the projects, we have modules for creating seeds defined in lib as a regular .ex file because we want these modules to be available in release. We run them when we launch a new server for testing. This happens quite often thanks to CI/CD. So, removing seeds.exs completely is OK, if someone is struggling with this just because the file is shipped with the default phoenix setup.

Post #5

fuelen

One tiny optimization could be applied with placeholders. Probably, it won’t be noticeable, but still good as an exercise.

Post #13

dimitarvp

Seeds should be IMO idempotent which means that you should have corresponding Repo.delete or Repo.delete_all before inserts. This of course assumes that you can in fact identify the data that must be seeded and remove it at all which in many cases is not possible i.e. I worked in places where certain tenants / companies / users must always exist and some even had hardcoded IDs. You can still achieve idempotency by simply not deleting and not inserting anything if you detect that these special records that must be always exist are already there.
I would not rely on the current datetime; I’d hardcode now as a fixed datetime in the past. To me determinism trumps almost all other concerns. However that too is not a hard rule because there are businesses where certain records are only valid if they’ve been created no more than f.ex. 6 months ago. So use your best judgement but still hardcode as much as you can if you can get away with it. Again – determinism.

Disagreed, seeds are practically their own universe and the only thing they share with the main app is the storage (in most cases a relational DB). It’s quite OK to override various runtime properties like logging in there. I even stuffed OTel tracing in one project’s seeds because they were taking mysteriously long (spoiler alert: one table was too big and the seeds scanned for records + did various levels of locking; we fixed it after).

Post #2

Last Post!

coen.bakker

coen.bakker:

Enum.chunk_every(book_ids, batch_size)
|> Task.async_stream(
  fn batch ->
    Enum.each(
      batch,
      &Repo.insert_all(Page, build_pages_for_book.(&1))
    )
  end,
  max_concurrency: max_concurrency
)

To avoid loose ends: This code snippet doesn’t work as intended because the async stream never actually runs.

Fixed version: xlp-book-club-API/example/priv/repo/seeds.exs at main · elixirland/xlp-book-club-API · GitHub

Post #22

Where Next?

View thread on forum (has 22 replies!)

phoenix

idiomatic-elixir

learning-elixir

seed

elixirland

Home Questions & Help>Questions

34 854 22

Last post

REPLY VIEW THREAD WITH 22 POSTS

Latest Phoenix Threads

Starter - composable setup workflows for Phoenix apps, built on Igniter

News>Announcing

Is this `seeds.exs` idiomatic Elixir?

coen.bakker

Is this `seeds.exs` idiomatic Elixir?

First Post!

LostKobrakai

Most Liked

fuelen

fuelen

dimitarvp

Last Post!

coen.bakker

Where Next?

Trending in Questions

Durable Workflow Execution (temporal alternative for elixir)

Looking for a Prawn-Like PDF Generation Library in Elixir

Simple/light way to self host microapps and services?

Using Phoenix.LiveView.TagEngine as an EEx.Engine is deprecated!

How to integrate Hologram with existing LiveView app?

How to do proper post-release steps

How to start Oban, but delay job processing?

Other Trending Topics

Dexter - A fast, full-featured Elixir LSP optimized for large codebases

Beam Bots - Resilient Robotics on the BEAM

Emerge & Solve - a GUI framework for Elixir

Emily - A new MLX-based backend for Nx

Elixir-lang.org redesign

Programming Nerves (self-published)

Latest Phoenix Threads

Questions & Help>Questions

Latest on Elixir Forum

Sponsor Spotlight

Our Sponsors

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

We're in Beta

Options