How do you push data to external APIs?

Questions / Help
,
1

How do you push data to external APIs?
In one of my side projects I want to index some data to Redis whenever a DB entry is created/updated/deleted (the same scenario is also applicable if the external API is a Elasticsearch/Algolia endpoint).

For this I am calling a function in my Redis module within my create/update db functions.

Since indexing does not need to happen synchronously, I thought about speeding up the e2e response time by wrapping the call to the Redis client in Task.async/1.

My question is now: What happens if a lot of DB entries are created? In this case it would be better to bundle all the data to be indexed and then send it over the wire within a single batch request.

Would you then reach for different modules like GenServers? Is Flow or Broadway a good use case for this?
Would you use Task.async in my scenario at all? Or do it with the Task.Supervisor module to add some additional resilience?
Also keeping in mind that you do not want to exhaust the external API.

Thoughts?

2

Do not use Task.async/1 as this spawns task outside of supervisor tree, use Task.Supervisor.async/{2,3}.

Yes, it would be better to do so.

Yes. Create cache that will run function to reindex data that will debounce/throttle data. It can be GenStage, but do not need. All you need could be achieved via :gen_statem.

I would never use Task.async/1 in production.

1 Like
3

I would use Task.async in tests, or in the REPL.

4

In production

5

Sorry I read op as asking if you would use Task.async in “any” situation, misread “my”.