I’ve currently the following use case:
I need to enable configurable filtering for stream data (e.g. filter out/change data incoming data by a specific identifier or similar). For specifying the rules I constructed a JSON format that gets parsed.
Now two ways came into my mind that could handle this.
- I’m using specific config files that generate modules on application start. That modules are then getting called in the pipeline inside my consumer module.
- The JSON configs are stored in a database and are getting loaded into a process on
start_linkin my consumer module that saves the filtering state.
Is there already some lib or “best practice” on how to handle such a UC?
I also think about what is more effective in terms of computational runtime.
I appreciate any tips and tricks or further information about this topic.