How to update from deprecated Supervisor.spec to new Supervisor.behaviour?

A.Server is not a GenServer, it’s a supervisor. There are places where “supervising” processes from another GenServer make sense (see parent library), but in a general fashion I’d suggest not using that unless you really need it. Why do you think you need a GenServer to supervise other processes?

The docs for supervisors do have a section for module based supervisors:

Minimally adapted to your code that would be:

defmodule A.Supervisor do
  # Automatically defines child_spec/1
  use Supervisor

  def start_link(init_arg) do
    Supervisor.start_link(__MODULE__, init_arg, name: __MODULE__)
  end

  @impl true
  def init(_init_arg) do
    children = [
      B # or more likely `B.Server`
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end
end

Because I have a stateful library A that will have to supervise another stateful library B (which it depends on). The reason for this being that an application will use A but doesn’t care how A operates. That’s A’s responsibility. If A needs a stateful or stateless library(ies) to do it’s job, is only A’s concern.

Also, aren’t Supervisor’s GenServers in general?
Sorry for the confusion.

So I know you picked up the “stateful library” name from my post in another topic, but this one was meant only for demonstrational purpose. It’s not really a term used by anyone and won’t generally help people understand.

No. Both are processes, but not every process is automatically a GenServer. Basically there are processes, which are bare bone (created with spawn) and then there are various otp behaviours, which run in processes, like :gen_server (GenServer being the elixir wrapper), :supervisor (again Supervisor) but also :gen_event, :gen_statem, …, which don’t have wrapper modules in elixir core.

Generally I’d suggest you start with one application here (I’ll just take :a). The callback module of :a usually is A.Application. start/2 within that module commonly starts the root supervisor A.Supervisor. This one is started just via Supervisor API, without an actual module backing it up (A.Supervisor is just for naming the process). That supervisor (like any other) can now have children, which (in the general case) either are supervisors by themselves or workers (genservers, genstatem processes, …). Within the child supervisors you then have again children and with that nesting you can build up a tree of processes, with workers at the leaves and the rest being supervisors.

How you want to compose those is imo quite well explained here: The Hitchhiker's Guide to the Unexpected

The only reason for moving things into separate applications is either some self-contained code/functionality shall be shared between multiple projects depending on it or if (e.g. in prod) those applications might not run within the same beam instance or not even on the same host.