Effective way to enable modular filtering pipeline

Questions / Help
, ,
1

Hello together,

I’ve currently the following use case:
I need to enable configurable filtering for stream data (e.g. filter out/change data incoming data by a specific identifier or similar). For specifying the rules I constructed a JSON format that gets parsed.

Now two ways came into my mind that could handle this.

  1. I’m using specific config files that generate modules on application start. That modules are then getting called in the pipeline inside my consumer module.
  2. The JSON configs are stored in a database and are getting loaded into a process on start_link in my consumer module that saves the filtering state.

Is there already some lib or “best practice” on how to handle such a UC?
I also think about what is more effective in terms of computational runtime.

I appreciate any tips and tricks or further information about this topic.

2

No one has an opinion on this topic?

3

Do you have multiple filters per id, or just a single filter for each?

4

Just a single filter for each id.

5

And the id is a field within the JSON payload?

6

If you mean by payload the rules configuration then yes.

7

Well I was considering something like psuedo code:

Stream
...
|> Stream.filter(
  fn payload ->
    filter_fun = config.get_filter(payload['id'])
    filter_fun(payload)
  end)

Where the config would have a mapping of id -> required function:

id_1: &Filters.filter_fun_1/1

However, I don’t know of an ‘official’ way to choose functions for filtering.

1 Like