While taking a look at Stream.scan, I have no idea on these trigger meaning about, since creating a new transaction inside existing transaction should do nothing.
Could you please write us a snippet code as well?
bcahya
aseigo
Hm .. I had expected something similar to possible from Elixir (I have more experience with Erlang than Elixir, and sometimes this trips me up .. :/) .. that is unfortunate. (and a reminder to always try the code out locally before commenting here … teaches me for attempting quick comments during breaks at the office :P)
Seems that way… the alternatives don’t seem great, either, though: having a transaction identifier that gets passed around (and easily lost, and not easy to detect when nesting ..)? a transaction represented by a “magic” process? Nothing particularly pretty pops out at me .. though it would be nice to have what you are doing included in Ecto as a nice-to-have,as yo note:
That would be awesome 
thanks in advance for your efforts!
0
aseigo
The transaction would move from wrapping the flow to being in the final Stream.each lambda. That function would add the queries (or lambdas) in a list (which would be the accumulator), and then whenever there were 10 (or whatever # of) queries in that list, start a transaction that runs the queries (or lambdas, as they would have captured their variables) and empty the accumulator for the next run.
1
andre1sk
mjadczak
We mentioned this above, this would indeed be possible but requires extra logic to switch between the two and roll back changes on any failures in the import, with a transaction we simply need to rollback if any errors occur and we’re back to a good state. It’s definitely one possible solution though!
0
bcahya
Great, finally we solved our issue using Stream.chunk, following with Stream.each. Bind this list with transaction
0
fishcakez
Ecto Core Team
TL;DR the short answer is no, the long answer is yes
There is not any concurrency when doing database calls, they block and are applied in strict order. Actually the calling process accesses the database connection socket directly (and not via a connection process). This is very efficient as it minimises copying and message passing, and garbage collection from one transaction does not effect other callers.
Transactions combined with pooling is a difficult problem. We need to ensure that a lock is held on a database connection while the transaction takes place and only the desired process(es) access that database connection, begin is always run first, commit/rollback is always called last, and the database connection is released eventually. We can provide these guarantees if the transaction occurs inside a single function call. Even in this situation the error handling is non-trivial. I think it is unrealistic to expect most users to handle this correctly and I would not be confident to handle this myself.
This means that only one process can access a process from the pool at a time and a transaction must be run side a single function call.
This approach is very difficult because :gen_server.enter_loop never returns and if the GenServer hibernates the call stack is thrown away - so the transaction would never be committed. Of course it is possible to wrap the enter_loop in a try and catch a :normal exit, just make sure never to hibernate.
As explained above I don’t think it is advisable to provide this. However you may have noticed that Ecto.Adapters.SQL.Sandbox provides a similar mechanism. A lock is acquired on a connection and this process (and possibly others) can access it many times. Of course we wouldn’t want the transaction/savepoint semantics of the sandbox. Fortunately this pool is powered at the low level by the DBConnection.Ownership pool, which does not apply the transaction/savepoint semantics. It would be possible to either use the ownership based pool, or build on it in a similar way to the sandbox does to provide guarantees on the transaction. I think the later would be preferable because it makes it easier to guarantee the transaction semantics. This could be achieved by copying the sandbox implementation and altering the ownership check in slightly such that there are 2 different ownership checkins, commit and rollback.
We would need to resolve this in DBConnection before Ecto. Pooling is separate from transaction handling there. Hopefully something related will appear before my Elixir Conf EU talk ;).
3
mjadczak
Thanks for a thorough and complete answer! It makes the situation much more clear.
This seems like the right solution for now, then. I’ll wrap this in a single module rather than making this extensible in any way to ensure that the GenServer does actually exit at some point, and that it does not hibernate (along with plenty of comments to ensure it stays that way in the future).
I see, seems like this thing runs all the way down. It may be worth reconsidering, but I think there should probably be other use-cases than just this, especially since it could be implemented with the solution above.
Thanks again!
0
fishcakez
Ecto Core Team
There is already work in progress on an alternative API for long running concurrent workflows that plays nicely with existing libraries.
1