mccraigmccraig

mccraigmccraig

Skuld - an effectful approach to deterministic testing, N+1 batching, and serialisable workflows

Skuld started out as a limited exploratory project, and then it snowballed…

Most Elixir apps have a layer of orchestration code bridging whatever pure model has been extracted with side-effecting components - “fetch user, check permissions, load subscription, hit an API, compute a price, write an invoice.” This code is tangled with databases, HTTP clients, and randomness. It’s hard to test, hard to refactor, and oftentimes impossible to property-test.

Skuld lets you write that code as pure descriptions of side effects. Handlers decide what those descriptions mean. Same code runs with real I/O in production, in-memory maps in tests, and properties you can verify with stream_data.

Here’s what that looks like:

Automatic N+1 batching. This reads like sequential code but fetch_user and fetch_orders run concurrently, and all the AccountQueries functions get batched into bulk queries. Stream 10 users with concurrency: 4 and FiberPool batches all deffetch calls into round-trips of [4, 4, 2]. No manual loader, no explicit batching:

defquery build_account_summary(user_id, month) do
  user <- AccountQueries.fetch_user(user_id)
  orders <- AccountQueries.fetch_orders(user_id, month)
  details <- Query.map(Enum.map(orders, & &1.id), &AccountQueries.fetch_order_details/1)
  build_account_summary(user, orders, details)
end

user_ids
|> Brook.from_enum()
|> Brook.map(&build_account_summary(&1, "2026-01"), concurrency: 4)
|> ...

Pausable state machines. A computation pauses at each Yield.yield, waits for input, resumes with full effect context. The same code wrapped in an AsyncCoroutine pauses for the user in a LiveView handle_info or, wrapped in a plain Coroutine, runs as three Coroutine.run calls in tests:

defcomp checkout do
  cart <- Yield.yield(:get_cart)
  {:ok, inventory} <- Inventory.check_stock(cart.items)
  payment <- Yield.yield(:get_payment)
  {:ok, order} <- Orders.place(cart, payment)
  ...
end

Durable computation. The same checkout state machine, above, but now serialised: pause mid-flight, save its entire execution history as JSON, resume after a restart. Every effect invocation is captured in a serialisable log. SerializableCoroutine.run(json, sc, cart) replays recorded effects and continues where it left off:

sc = SerializableCoroutine.new(checkout, fn comp -> ... end)
suspended = SerializableCoroutine.run(sc)
json = SerializableCoroutine.serialize(SerializableCoroutine.get_log(suspended))
# => EffectLogEntry{data: :get_cart, value: nil, state: :started}

SerializableCoroutine.run(json, sc, %{items: [...]})
# => EffectLogEntry{data: :get_cart, value: %{items: [...]}, state: :executed}
# => EffectLogEntry{data: :get_payment, value: nil, state: :started}

Under the hood there are algebraic effects, but you don’t need to know that. Writing domain code with State, Reader, Yield, Port (for external dispatch), and Repo (for database operations) reads naturally once you’ve seen the examples.

The library bundles effects for state management, cooperative concurrency (FiberPool, Channel, Brook), value generation (Fresh, Random), error handling (Throw, Bracket), external integration (Port, Repo, hexagonal architecture), and serialisable workflows (EffectLogger, SerializableCoroutine). Full docs at hexdocs.pm/skuld.

Add {:skuld, "~> 0.28"} to your deps.

I’d love to hear what you think - so far I’ve had reactions varying from solid interest to “I would burn that with fire”!

https://github.com/mccraigmccraig/skuld

Most Liked

mccraigmccraig

mccraigmccraig

I could have talked about backpressure… that works very nicely with Brook … but I thought the automatic batching/N+1 elimination was more fun!

Where Next?

Popular in Announcing Top

KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36654 110
New
jakub-zawislak
Hi everyone, I’m coming from the Symfony (PHP) framework. I like Phoenix, but it has a one thing that was build much better in the Symfo...
New
Crowdhailer
Raxx is an alternative to Plug and is inspired by projects such as Rack(Ruby) and Ring(Clojure). 1.0-rc.1 is now available. To use it re...
New
anshuman23
Hello all, I have been working on my proposed project called Tensorflex as part of Google Summer of Code 2018.. Tensorflex can be used f...
New
mindok
What is ContEx? A pure Elixir server-side data plotting/charting library outputting SVG. It has nice barcharts in particular and works g...
New
mplatts
With HEEX released we decided to start a components library using Tailwind CSS - check it out here: Petal Components. We also have a boi...
New
wojtekmach
Hey everyone! Req is an HTTP client for Elixir that I’ve been working on for quite some time. There is already a lot of HTTP clients out...
New

Other popular topics Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New

We're in Beta

About us Mission Statement