I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records approximately (15MB).
Each record in the table has a email message id with which I have to call Google Gmail API to get the message.
So I am thinking about memory efficient. What is the most efficient way to call all the records with the minimum impact in memory or not loading the hole 15MB at once just to make API calls?
I found Ecto Stream but it works inside a transaction and I don’t use transactions. And… is Ecto Stream the best way to go?
Thanks for any comments.