mccraigmccraig

mccraigmccraig

Skuld - an effectful approach to deterministic testing, N+1 batching, and serialisable workflows

Skuld started out as a limited exploratory project, and then it snowballed…

Most Elixir apps have a layer of orchestration code bridging whatever pure model has been extracted with side-effecting components - “fetch user, check permissions, load subscription, hit an API, compute a price, write an invoice.” This code is tangled with databases, HTTP clients, and randomness. It’s hard to test, hard to refactor, and oftentimes impossible to property-test.

Skuld lets you write that code as pure descriptions of side effects. Handlers decide what those descriptions mean. Same code runs with real I/O in production, in-memory maps in tests, and properties you can verify with stream_data.

Here’s what that looks like:

Automatic N+1 batching. This reads like sequential code but fetch_user and fetch_orders run concurrently, and all the AccountQueries functions get batched into bulk queries. Stream 10 users with concurrency: 4 and FiberPool batches all deffetch calls into round-trips of [4, 4, 2]. No manual loader, no explicit batching:

defquery build_account_summary(user_id, month) do
  user <- AccountQueries.fetch_user(user_id)
  orders <- AccountQueries.fetch_orders(user_id, month)
  details <- Query.map(Enum.map(orders, & &1.id), &AccountQueries.fetch_order_details/1)
  build_account_summary(user, orders, details)
end

user_ids
|> Brook.from_enum()
|> Brook.map(&build_account_summary(&1, "2026-01"), concurrency: 4)
|> ...

Pausable state machines. A computation pauses at each Yield.yield, waits for input, resumes with full effect context. The same code wrapped in an AsyncCoroutine pauses for the user in a LiveView handle_info or, wrapped in a plain Coroutine, runs as three Coroutine.run calls in tests:

defcomp checkout do
  cart <- Yield.yield(:get_cart)
  {:ok, inventory} <- Inventory.check_stock(cart.items)
  payment <- Yield.yield(:get_payment)
  {:ok, order} <- Orders.place(cart, payment)
  ...
end

Durable computation. The same checkout state machine, above, but now serialised: pause mid-flight, save its entire execution history as JSON, resume after a restart. Every effect invocation is captured in a serialisable log. SerializableCoroutine.run(json, sc, cart) replays recorded effects and continues where it left off:

sc = SerializableCoroutine.new(checkout, fn comp -> ... end)
suspended = SerializableCoroutine.run(sc)
json = SerializableCoroutine.serialize(SerializableCoroutine.get_log(suspended))
# => EffectLogEntry{data: :get_cart, value: nil, state: :started}

SerializableCoroutine.run(json, sc, %{items: [...]})
# => EffectLogEntry{data: :get_cart, value: %{items: [...]}, state: :executed}
# => EffectLogEntry{data: :get_payment, value: nil, state: :started}

Under the hood there are algebraic effects, but you don’t need to know that. Writing domain code with State, Reader, Yield, Port (for external dispatch), and Repo (for database operations) reads naturally once you’ve seen the examples.

The library bundles effects for state management, cooperative concurrency (FiberPool, Channel, Brook), value generation (Fresh, Random), error handling (Throw, Bracket), external integration (Port, Repo, hexagonal architecture), and serialisable workflows (EffectLogger, SerializableCoroutine). Full docs at hexdocs.pm/skuld.

Add {:skuld, "~> 0.28"} to your deps.

I’d love to hear what you think - so far I’ve had reactions varying from solid interest to “I would burn that with fire”!

Most Liked

mccraigmccraig

mccraigmccraig

I could have talked about backpressure… that works very nicely with Brook … but I thought the automatic batching/N+1 elimination was more fun!

Where Next?

Popular in Announcing Top

wmnnd
Hi there, for my project DBLSQD, I needed a file storage solution that is a bit more flexible than Arc. Because I thought others might f...
New
Crowdhailer
The latest release of Ace (0.10.0) includes serving content over HTTP/2. I have started writing a webserver to teach my self more about...
New
mikehostetler
I’m excited to announce Jido, a framework providing foundational primitives for building autonomous agent systems in Elixir. While develo...
New
mspanc
I am pleased to announce an initial release of the Membrane Framework - an Elixir-based framework with special focus on processing multim...
New
alisinabh
Hey everyone i’ve developed a library for Jalaali calendar for elixir which supports converting Gregorian dates to Jalaali and vice vers...
New
Qqwy
Hello everyone, I wrote a small library today called MapDiff. It returns a map listing the (smallest amount of) changes to get from map...
New
RobertDober
Earmark is a pure-Elixir Markdown converter. It is intended to be used as a library (just call Earmark.as_html), but can also be used as...
239 12512 134
New
treble37
Just looking for a little feedback on a tiny helper library I built - Sometimes I find the need to convert maps with atom keys to maps...
New
sasajuric
I’d like to announce a small library called boundaries. This is an experimental project which explores the idea of enforcing boundaries ...
New
scohen
Lexical Lexical is a next-generation language server for the Elixir programming language. Features Context aware code completion As-you...
New

Other popular topics Top

sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 42842 311
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31107 143
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New

We're in Beta

About us Mission Statement