I’ve an oracle database with a little over 18m records. I need to fetch these records and run them against a business logic. What will be the best approach - Can a genstage/flow help in this context.
I plan creating a process that will fetch a count of all the records, then start about 20 processes that will fetch records concurrently using limit/row_number then process the records.
4 Likes
If you have a specific question feel free to follow up but generally, yes, this sounds like a fantastic application of Flow in particular. Broadway would be best suited but there’s certainly not an oracle db Producer for that yet so that seems premature. And by the time you come up with your own producer you could probably implement a chain of Flow commands to do what you want.
In terms of interfacing with Oracle, I’m assuming folks are still either using some community erlang libraries or using some kind of Postgres wrapper for Oracle.
I played with both oracle_fdw and jamdb a few months ago. IMHO, jamdb is easier to use. It’s not a full Oracle driver implementation, but the basic functionality is all there.
Some of us would really appreciate if you let us know about the results of this activity.
Good luck
EDIT: Even when it just mentions Erlang in its description, jamdb works fine with Elixir as well.
3 Likes
Flow is kind of new to me, came across it recently when going through elixir stuff on youtube. I will read further and give it a shot. With Flow; do I have to create several Flow processes each with it own set of data to work on. How will the partition function work in this case.
I will do a POC with the two and see how it goes. I will keep you posted with my findings.
1 Like
Excellent. Thank you.
Also, there’s another Erlang/Elixir driver for Oracle:
I didn’t try it because it hasn’t been updated in two years, and it’s based on ODBC which I thought was going to be slower than the other two alternatives.
Cheers
1 Like
