Absinthe.Subscription.publish performance

Hi @aloukissas publish is indeed expensive by design, it is not instant like broadcast. Unlike broadcast which merely needs to send a message to various processes, Absinthe needs to execute the documents associated with a topic before it can send results to end users. So some process somewhere needs to bear the cost of performing this execution.

By default, that is the process that triggers the publish call, generally a mutation. This operates as a form of back pressure, where load incurred by an operation on the system is felt by the request that triggers that load.

Thanks for confirming Ben!

Should we expect to see better performance on the mutation if instead of doing out-of-band call to publish, we use the trigger functionality like in the example in the docs?

No trigger just runs the publish call inside of middleware on the mutation same as calling publish. You can spawn a task and call publish inside the task, or (better) you could build like an oban job queue of publish calls perhaps. The main danger to just spawning tasks is that you overload and potentially even OOM the system if you have many subscribers and many mutations on those subscribers without any backpressure.

Following up here: we are noticing that Absinthe.Subscription.Local.publish_mutation/3 can often be very spiky wrt latency, sometimes hitting > 1 second. The resolvers involved are instantaneous: they use the provided payload and optionally Nebulex in-memory cache (in the microseconds).

Here’s the interesting part: we see huge gaps in the traces between when the publish_mutation is called until the resolvers are called (i.e. when the [:absinthe, :subscription, :publish, :start] and [:absinthe, :subscription, :publish, :stop] telemetry events are emitted. This span can e.g. take 50 msec yet the entire publish/3 span can take 10-20x that, in unaccounted for time.

If I’m reading the code correctly, the code below is what’s run before (gh link):

      for {field, key_strategy} <- subscribed_fields,
          {topic, doc} <- get_docs(pubsub, field, mutation_result, key_strategy) do
        {topic, key_strategy, doc}
      end

This calls Registry module underneath.. I wonder if under pressure this can have performance issues?

@aloukissas not sure if this is applicable in your case, or you can make it applicable, but I found a tremendous performance improvement after deduplication subscription resolvers Understanding Subscriptions — absinthe v1.11.0

Subscriptions by default are expensive because they have to be resolved in context of each connected user. You can pass a context to it and make it resolve only once.

That not always can be applied, obviously if different users should see different data it often can’t work.

I learned this the hard way when I first started using Absinthe with subs. Dedup should be the default, things don’t really work at all without it.

That’s a broader topic about GraphQL in general and subscriptions share the same possible negative implications for performance as the protocol in general.

I kinda use them as a pubsub mechanism these days, and use it to notify that something changed rather than sending the big payload in subscription. That helps too, especially if the client then can batch up debounce their own API calls to fetch the updated data.

If you keep the subscriptions simple, and keep the Graph tree flat / two levels max with no loops, GraphQL is pretty nice.

TBH, using subs as a signaling system is wrong. There is something in absinthe subscriptions that does not work well under load. We’ve forked it and added a ton more instrumentation to help us identify these.

Hey @aloukissas! Did you find anything useful through your instrumentation?

I have found this comment about context size that could be relevant, but haven’t started digging too deep yet: Absinthe cannot seem to handle even 1000 concurrent subscriptions - #2 by josevalim