fklement
Effective way to enable modular filtering pipeline
Hello together,
I’ve currently the following use case:
I need to enable configurable filtering for stream data (e.g. filter out/change data incoming data by a specific identifier or similar). For specifying the rules I constructed a JSON format that gets parsed.
Now two ways came into my mind that could handle this.
- I’m using specific config files that generate modules on application start. That modules are then getting called in the pipeline inside my consumer module.
- The JSON configs are stored in a database and are getting loaded into a process on
start_linkin my consumer module that saves the filtering state.
Is there already some lib or “best practice” on how to handle such a UC?
I also think about what is more effective in terms of computational runtime.
I appreciate any tips and tricks or further information about this topic.
Most Liked
mpope
Well I was considering something like psuedo code:
Stream
...
|> Stream.filter(
fn payload ->
filter_fun = config.get_filter(payload['id'])
filter_fun(payload)
end)
Where the config would have a mapping of id → required function:
id_1: &Filters.filter_fun_1/1
However, I don’t know of an ‘official’ way to choose functions for filtering.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Enabling companies to succeed by building software people love.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"