I want to use Repo.stream to process a number of records, per my understanding it must be wrapped within Repo.transaction. I was wondering if there’s a way or another function which will allow usage of Repo.stream and partially commit transaction unlike Repo.transaction that will commit when all transaction are successful or rollback otherwise - like having a savepoint in the transaction.
You can implement savepoints and partial rollbacks inside of a transaction, but you cannot partially commit data in a transaction.
Repo.stream uses a cursor under the hood, which must be declared in an existing transaction.
If we put these together, then we will conclude that you cannot commit inside of a Repo.stream until the full execution is done. You could probably put something together with a multi-process approach (and 2 active transactions), but you are getting into challenges with the MVCC model at that point.
I’m well-versed in postgres, but haven’t used cursors that much. If anyone sees this post as incorrect, please correct me!