{:error, reason} is an unexpected return value for the init callback, the possible return values are listed in the GenServer docs:

Returning {:ok, state} will cause start_link/3 to return {:ok, pid} and the process to enter its loop.

Returning {:ok, state, timeout} is similar to {:ok, state} except handle_info(:timeout, state) will be called after timeout milliseconds if no messages are received within the timeout.

Returning {:ok, state, :hibernate} is similar to {:ok, state} except the process is hibernated before entering the loop. See handle_call/3 for more information on hibernation.

Returning :ignore will cause start_link/3 to return :ignore and the process will exit normally without entering the loop or calling terminate/2. If used when part of a supervision tree the parent supervisor will not fail to start nor immediately try to restart the GenServer. The remainder of the supervision tree will be (re)started and so the GenServer should not be required by other processes. It can be started later with Supervisor.restart_child/2 as the child specification is saved in the parent supervisor. The main use cases for this are:

The GenServer is disabled by configuration but might be enabled later.
An error occurred and it will be handled by a different mechanism than the Supervisor. Likely this approach involves calling Supervisor.restart_child/2 after a delay to attempt a restart.
Returning {:stop, reason} will cause start_link/3 to return {:error, reason} and the process to exit with reason reason without entering the loop or calling terminate/2.

I think you can achieve your desired result by returning {:stop, reason} from your init callback:

def init(list) do
  case Registry.register(MyCoolRegistry, list[:key], :ok) do
    {:ok, _} -> {:ok, list}
    {:error, reason} -> {:stop, reason}
  end
end