Hello! I’m building an application using Commanded with the Extreme EventStore Adapter and Ecto Projections and i’m having a difficulty to run it in a clustered manner.
By default, when i create a projection the Persistent Subscription on EventStore is created with the limit of 1 subscriber per subscription, in that way, after the first node connection to the eventstore, every other node will throw a subscription_already_exists error.
I took a look on the documentation for extreme and extreme adapter and i have not found a way to increase this number, i took a look at the code and found that the SubscriptionsSupervisor for the Extreme adapter has the option Keyword.get(opts, :subscriber_max_count), but i have not found any way to set this value from the Projection or through the config file.
Does anyone know how can i increase this number?
Thanks in advance!
1 Like
I’ve added an issue to the GitHub repo for the Commanded extreme adapter to allow configuration of the subscriber max count for persistent subscriptions.
The limit is set to 1 by default to ensure that only one event handler can be subscribed at a time. This is because event handlers usually want to process events in order. As an example, Commanded Ecto projections uses the event number for idempotency checking to ensure that events received more than once, due to at-least-once-message delivery, will only be projected once. If you were to have the same projection process running on multiple nodes subscribed to the same persistent subscription then events would be processed concurrently and likely out of order. The idempotency check would mean that any out of order events would be skipped because they would be assumed to have already been processed. The current approach is to have a single connected subscriber which guarantees event ordering.
In a multi-node deployment with distributed Erlang the :global module is used to guarantee that only one event handler process will be running within the cluster. This ensures only a single persistent subscription is connected and guarantees event ordering. If you don’t use distributed Erlang then you will have an event handler process started on each node. Unfortunately the extreme library doesn’t handle the case where connecting to a persistent subscription fails due to too many subscribers. This is a limitation that needs to be addressed. It could implement some retry mechanism when connecting to the subscription fails which would allow multiple handlers to be started but only one would connect and actually process the events. This is how the Elixir EventStore using Postgres is implemented.
2 Likes
Good Morning!
Thank you for your amazing reply, for now i changed from Swarm to Erlang Global Module and it should work for what i need in the moment, thank you!
