Using external HTTP API for data storage

If it were me I’d start with the local database and work toward eventual consistency with the external system. There are places where it can be a gatekeeper like user registration but it does make less sense if the external system is reliable, just quirky. A local system would be useful during outages but that could be extremely rare too. It’s certainly more complex because the lines of separation blur easily but you can have an eye on being platform agnostic. Companies get acquired or shutter or grow to be unicorns so all of this suggestion could also be a premature optimization.

Ecto and its family seem to map more to database like concepts. Cloud type data systems using http APIs certainly exist but they are likely closer to the primitives Ecto exposes than something more CRUD like. I also suffer from limited thinking so an adapter could very much pave the way for doing this between multiple systems. I feel like a generic adapter could be too ambitious without trying this between 3-5 systems to sus out the common bits.

I also can’t adequately evaluate which of just these two is more complex. Ecto potentially has a risk of not paying off at all where eventually consistent seems to have less of that risk, you just may throw more time and frustration at it.

In a perfect situation I’d spike both and leave myself open for a third option or a hybrid. There may be resources that must live local with others that can be more remote-first.

You can use Ecto embedded changesets for this, simply define all your CRUD changesets and validation in Ecto as you normally would.

Your contexts decide what to do with those changesets, how to read, create, update, and delete, just follow the same pattern of changeset validation, and pipe to an external API instead of Ecto.Repo. Remember that Ecto.Repo interfaces with an external system too as it maintains a connection pool to an external database.

You will probably want to think about a connection pool for the API. Here is a tutorial of following the similar model that Ecto uses for maintaining a pool using poolboy.

If you are using a http interface then req which is built on top of finch/mint/nimblepool also has pools integrated, the nice part being finch/mint/nimblepool is already included with phoenix.

If you are using the API at a rudimentary level with your own http request handling I would create a thin abstraction around around it to handle auth token refresh (something req can help with too with a retry error step) and mapping errors to changeset field errors and so on and this will make it easier to test also.

I kinda stopped exploring the ideas around EctoApi bc I felt the db approach to an API is very brittle, … I went in another direction and started boto it’s very early stage, i had to stop my experiments there bc I was dealing with some more urgent projects at work and also i’m not with that much free time to finish the adjustments. I might restart working on boto next year.

I personally think it’s very brittle so it would either be very limited or at least very unpredictable.
A sign of that is projects that go the other way around and try to expose sql functionalities to a rest api, like sqlrest that doesn’t get that much of a traction.

I think that’s the best approach if you want to avoid overengineer the problem.