How do you push data to external APIs?
In one of my side projects I want to index some data to Redis whenever a DB entry is created/updated/deleted (the same scenario is also applicable if the external API is a Elasticsearch/Algolia endpoint).
For this I am calling a function in my Redis module within my create/update db functions.
Since indexing does not need to happen synchronously, I thought about speeding up the e2e response time by wrapping the call to the Redis client in Task.async/1.
My question is now: What happens if a lot of DB entries are created? In this case it would be better to bundle all the data to be indexed and then send it over the wire within a single batch request.
Would you then reach for different modules like GenServers? Is Flow or Broadway a good use case for this?
Would you use Task.async in my scenario at all? Or do it with the Task.Supervisor module to add some additional resilience?
Also keeping in mind that you do not want to exhaust the external API.