If I got that correctly, your app uses an external API to fetch data, transforms that into your own format, saves that in DB and then exposes that through your own API. The problem is that the external API is really unreliable. As a consequence, you don’t really have an option to implement it via hot calls.

So what I’m suggesting is that you “clone” the external database. You have options here:

1. dump the responses from the API straight into the DB: the advantage here is that you can easily change your own JSON schema (add more attributes, etc.), since the external data sits in your DB
2. store the transformed entities in the DB: the advantage here is speed (negligible)
3. do both

Seems like the problem that you need to solve here is to decide what data and when to pull it from the external API.

As to what:

• can you store everything locally? what’s the estimated data set? can you afford this?
• does your application need all the data?

As to when:

• does the external API has web hooks or any other notification mechanism?
• does the external API has an incremental endpoint (i.e. give me what changed since timestamp)?
• if you need to poll for data, what’s the acceptable timespan? how is this affected by quota?

Depending on the answers, the whole thing might event turn infeasible (I hope not!).

I’d implement this export as a worker that runs continuously in the background under it’s own supervision tree. As to how organize this - it really depends on the structure of the data and what the external API gives you. But my first approach would be a process that has a queue of messages, where each message represents a resource to be fetched. For each message, a task is spawned to fetch the data. If it fails, the message is resent to try again. Some additional process would also periodically request fetching of all resources.

you can do retries with tesla
https://github.com/teamon/tesla

the missing data part, sounds domain specific so I think you would have to roll your own, maybe against a changeset, and if it’s invalid you load in the missing data from the api..

for the cron part you can use quantum GitHub - c-rack/quantum-elixir: Cron-like job scheduler for Elixir · GitHub

I would consider making a mirror of the external system, i.e. dump the data (JSON, XML, whatnot) into your database as is; put that into a worker and handle external API failures there.. Then add a separate worker that would transform and upload the data to the destination. This might make for a cleaner design and better performance.