Fuse (circuit breaker) not breaking when called asynchronously

Questions / Help
1

I’m getting some unexpected results when using fuse.
Synchronous calls behave as expected unlike async calls.
I am trying to understand what is going on.
Basically this is what I’m doing:

defmodule App.Fuse do
  alias __MODULE__
  @name :fuse_name
  @options {{:standard, 2, 10_000}, {:reset, 60_000}}
  
  def exec(func) do
    run = fn ->
      case func.() do
        {:ok, result} ->
          {:ok, {:ok, result}}
        error ->
          {:melt, error}
      end
    end
    case :fuse.run(@name, run, :sync) do
      {:ok, result} ->
        {:ok, result}
      :blown ->
        {:error, :blown}
      {:error, :not_found} ->
        install_fuse()
        exec(func)
    end
  end

  def install_fuse, do:
    :fuse.install(@name, @options)

  def ask_status, do:
    :fuse.ask(@name, :sync)

  defmodule Client do

    def call_synchronous do
      generate_calls()
      |> Enum.map(&run_in_fuse_context/1)
      |> Enum.map(& &1.())
      |> (fn m -> {Fuse.ask_status(), m} end).()
    end

    def call_async do
      generate_calls()
      |> Enum.shuffle
      |> Enum.map(&run_in_fuse_context/1)
      |> Enum.map(&Task.async/1)
      |> Enum.map(&Task.await(&1, 30_0000))
      |> (fn m -> {Fuse.ask_status(), m} end).()
    end

    defp run_in_fuse_context(c), do:
      fn -> Fuse.exec(c) end
  
    defp run_something(idx, 0), do:
      {:error, idx}
  
    defp run_something(idx, _), do:
      {:ok, idx}
  
    defp generate_calls do
      # no concurrent :fuse.install
      Fuse.install_fuse()
      # produce :ok, :ok, :error, .., and so on
      Enum.map(1..20, fn idx ->
        fn -> run_something(idx, rem(idx, 3)) end
      end)
    end
  end

end

Synchronous result:

iex> App.Fuse.Client.call_synchronous
{:blown,
 [
   ok: {:ok, 1},
   ok: {:ok, 2},
   ok: {:error, 3},
   ok: {:ok, 4},
   ok: {:ok, 5},
   ok: {:error, 6},
   ok: {:ok, 7},
   ok: {:ok, 8},
   ok: {:error, 9},
   error: :blown,
   error: :blown,
   error: :blown,
   error: :blown,
   error: :blown,
   error: :blown,
   error: :blown,
   error: :blown,
   error: :blown,
   error: :blown,
   error: :blown
 ]}

Async result:

iex> App.Fuse.Client.call_async
{:blown,
 [
   ok: {:ok, 17},
   ok: {:ok, 11},
   ok: {:ok, 10},
   ok: {:error, 18},
   ok: {:ok, 20},
   ok: {:ok, 1},
   ok: {:error, 9},
   ok: {:error, 3},
   ok: {:ok, 14},
   ok: {:ok, 4},
   ok: {:ok, 7},
   ok: {:ok, 2},
   ok: {:error, 15},
   ok: {:error, 6},
   ok: {:ok, 13},
   ok: {:ok, 8},
   ok: {:ok, 5},
   ok: {:error, 12},
   ok: {:ok, 16},
   ok: {:ok, 19}
 ]}

So after all of the async calls are done, the fuse is blown.
Any ideas on why the circuit doesn’t break “somewhere in the middle” ?

2

@i-n-g-m-a-r It does break in the middle, it breaks in the middle of the execution order, not in the middle of the list. You’re running each thing asynchronously, so the execution order will not be the same thing as the list order, that’s just the nature of running a bunch of things concurrently.

3

Tnx @benwilson512 for trying to explain.
I think I understand what you mean by execution order versus list order.
Maybe I was expecting something like this as a result of the async call:

iex> App.Fuse.Client.call_async
{:blown,
 [
   error: :blown,
   ok: {:ok, 11},
   ok: {:ok, 10},
   ok: {:error, 18},
   error: :blown,
   ok: {:ok, 1},
   ... etc ...
 ]}

So including a bunch of {:error, :blown} results but not in any particular order.
But I guess the run_something functions were already allowed to execute.

4

Ah, so the only way you’d get exactly 3 failures is if :fuse forced everything to run one after another so it could count perfectly. This would have really bad performance implications though. If I had to guess, :fuse implements the counters asynchronously, so if you have a ton of work that all happens basically instantly and at the same time, some will happen before the counter has caught up and the fuse is blown.

1 Like
5

It’s probably also worth mentioning that circuit breakers are not really meant to be calculating perfectly. They’re kill switches to prevent continuous overload from happening, but not perfect rate limiters, which prevent any calls beyond the threshold instantly. You could make the calls to fuse sync iirc, but as mentioned it’ll hurt performance.

2 Likes
6

Tnx for the comments!
The results make sense now.