coen.bakker
Is this `seeds.exs` idiomatic Elixir?
One of the requirements of the Book Club exercise on elixirland.dev is to seed data.
I am referring to this requirement in the exercise.
## Requirements
...
### **Seeding**
* Running mix ecto.setup creates the database tables but also seeds the database
* Seeding inserts 4,000 books that each have 10 pages
* Some seeded books have an active page, but not all
* Seeding is fast
How happy or unhappy would you be if a co-worker would create a pull request containing this seeds.exs? Is it written in idiomatic Elixir? Use of comments, etc.
import Ecto.Query, only: [from: 2]
alias BookClub.Repo
alias BookClub.Books.{Book, Page}
# Setup
# Sets log level to :info for performance
initial_log_level = Logger.level()
Logger.configure(level: :info)
IO.puts("Start database seeding")
start_time = System.os_time(:millisecond)
# Constants
n_books = 4000
n_pages_per_book = 10
inserted_at = NaiveDateTime.utc_now(:second)
batch_size = 800
# Insert 4,000 books
books =
for _ <- 1..n_books do
%{
title: XlFaker.generate_title(),
inserted_at: inserted_at,
updated_at: inserted_at
}
end
books
|> Enum.chunk_every(batch_size)
|> Enum.each(&Repo.insert_all(Book, &1))
# Batch insert 10 pages per book
# Gives some books an active page
book_ids =
from(b in Book, select: b.id)
|> Repo.all()
pages =
book_ids
|> Enum.flat_map(fn book_id ->
pages =
for i <- 1..n_pages_per_book do
%{
book_id: book_id,
content: XlFaker.generate_page(),
number: i,
status: :inactive,
inserted_at: inserted_at,
updated_at: inserted_at
}
end
List.update_at(
pages,
:rand.uniform(n_pages_per_book) - 1,
&Map.put(&1, :status, Enum.random([:active, :inactive]))
)
end)
pages
|> Enum.chunk_every(batch_size)
|> Enum.each(&Repo.insert_all(Page, &1))
# Teardown
end_time = System.os_time(:millisecond)
IO.puts("Finish database seeding")
IO.puts("Seeded #{n_books} books in #{end_time - start_time}ms")
Logger.configure(level: initial_log_level)
Most Liked
fuelen
I’d be okay if this script is just a temporary solution. The code itself is good enough. My concerns are not about style of Elixir code but more about approach in general. It works for simple cases, but as the project grows, I’d not put
* Seeding is fast
to the requirements.
What is more important for me is correctness of data. The approach of inserting raw data will quickly become a mess on several dozen tables. I’m a proponent of using business functions in seed scripts. I’m okay to wait a bit longer. Inserting seed data can be sped up by parallelization (Task.async_stream or flow can help). Because of that, having progress bars in seeds is also a good thing (owl can help).
The number of inserted data must be configurable, so it is possible to specify tiny numbers to run seeds as a part of the test suit. Just create 1 record for each record type to ensure that the script doesn’t fail. Most likely, you won’t be happy when the script stops working when you need it the most 
On one of the projects, we have modules for creating seeds defined in lib as a regular .ex file because we want these modules to be available in release. We run them when we launch a new server for testing. This happens quite often thanks to CI/CD. So, removing seeds.exs completely is OK, if someone is struggling with this just because the file is shipped with the default phoenix setup.
fuelen
One tiny optimization could be applied with placeholders. Probably, it won’t be noticeable, but still good as an exercise.
dimitarvp
-
Seeds should be IMO idempotent which means that you should have corresponding
Repo.deleteorRepo.delete_allbefore inserts. This of course assumes that you can in fact identify the data that must be seeded and remove it at all which in many cases is not possible i.e. I worked in places where certain tenants / companies / users must always exist and some even had hardcoded IDs. You can still achieve idempotency by simply not deleting and not inserting anything if you detect that these special records that must be always exist are already there. -
I would not rely on the current datetime; I’d hardcode
nowas a fixed datetime in the past. To me determinism trumps almost all other concerns. However that too is not a hard rule because there are businesses where certain records are only valid if they’ve been created no more than f.ex. 6 months ago. So use your best judgement but still hardcode as much as you can if you can get away with it. Again – determinism.
Disagreed, seeds are practically their own universe and the only thing they share with the main app is the storage (in most cases a relational DB). It’s quite OK to override various runtime properties like logging in there. I even stuffed OTel tracing in one project’s seeds because they were taking mysteriously long (spoiler alert: one table was too big and the seeds scanned for records + did various levels of locking; we fixed it after).
Last Post!
coen.bakker
To avoid loose ends: This code snippet doesn’t work as intended because the async stream never actually runs.
Fixed version: xlp-book-club-API/example/priv/repo/seeds.exs at main · elixirland/xlp-book-club-API · GitHub
Popular in Questions
Other popular topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"