I’m working on a project one part on which requires information (for example, external systems resource consumption info) from one or many external sources of the same type. These external systems don’t have push or subscription options so I need to poll them periodically. The tricky part is that users must be able to add new or delete already added sources at any time as well as to change connection properties (like url, auth info, polling interval and so on). It is possible that in the future it will be possible to get information on subscription scheme.
This particular application’s part is a kind of connector\translator events from the external systems to the core domain. I’m going to use Commanded on the whole project.
I want to put this part into a Connectors subdomain and describe the Connection aggregate (names are debatable). I want to start polling right after this aggregate creation and stop on deletion.
So the question is what is the best way to start the polling processes?
In general it could be a GenServer process (as I need to keep some service information like the last success call timestamp, prevent parallel connections and so on) with poll request scheduling. I evaluate Commanded’s process manager but is it a good idea to make external calls right from it (through ports\adapters, of course, but still)? Or is it better to start an additional process via the same process manager on ConnectorAdded\ConnectorResumed events and stop it on ConnectorDeleted\ConnectorPaused?
UPD:
Or is it better to disconnect the polling part at all and use with the read model part (like on start read current active connectors list, start supervised processes and update them on new events)? I don’t know what would happened with each Process Manager on node restart: of course they are going to be restored but how to start scheduling\polling processes after that? Do I understand it correctly that process manager waits for particular events to become active and there is no standard way to resume it on restart?
Without knowing more about your system, I would question the use of ES at all.
This sounds – again, I don’t know what your use-cases are – this a simple read model and a GenServer that would resume on start.
If you really want to employ ES for this domain, I would register an EventHandler that listens for the pertinent events, update the read-model, and simply send messages to your GenServer.
Hmm… I think that @factoryd is on to something here perhaps. Greg Young warns on a few occasions “please dont tell me that you’ve built an event sourced system”, meaning that it only makes sense applying the pattern to a small part of your system where it’s actually really needed, not least because ES comes with a ton of overhead. There are actually quite a few wartime stories about botched ES attempts, and a common theme is attempting to force ES everywhere.
The reason I’m saying this is that it looks like you should perhaps think about separating your app into 2 parts: the first should only concern itself with the polling aspect (configuration screens, external apis etc.) that will produce commands that will then be pushed on to the second part of your system which should only be concerned with accounting (your core business, which may actually benefit from ES), that will simply consume the commands from the first part, convert those into events, apply them and finally publish them for the rest of the system to act upon (you know, the canonical ES flow).
Again, that would be my advice: do not mix a ton of different concerns in an “event sourced application” because this pattern tends to multiply the effects of bad system design decisions
Probably I wasn’t clear enough: my question was about this particular bounded context only (external data manipulation business logic and actual interface\port mechanics), not the whole (billing\accounting) system.
I don’t think it improves my original question anyhow (probably adds another ones ), but I consider this subdomain as supportive for the core domain. It should interact with user [configuration] commands, keep current connectors configuration, get external events, save them internally for the future needs (in case I’ll need event data parts I don’t currently use), then translate and populate to the core domain. There are probably more than 2 parts: configuration, actual external system connectors and translation logic.
For sure I try to be aware of possible problems and have read about them. ES has its own pros and cons, but every technology, pattern or idea has. I have no any practical experience with CQRS\ES but isn’t it the reason to get some, especially in case of internal project with my own resources involved only?
OK, I see. Perhaps it’s worth a shot then as a learning experience! But if this solo/personal/internal project ends up being used everywhere (as infrastructure projects are sometimes wont to be) you should make sure that you will be given the required time (by your manager for example) to revisit your designs in case they end up not fitting the problem
), but I consider this subdomain as supportive for the core domain. It should interact with user [configuration] commands, keep current connectors configuration, get external events, save them internally for the future needs (in case I’ll need event data parts I don’t currently use), then translate and populate to the core domain. There are probably more than 2 parts: configuration, actual external system connectors and translation logic.
