elnewbee
Design elixir application in OTP terms.
Greetings, everyone! 
When learning elixir, I faced up with the next task.
I have mongodb with stored entities that have id, created_at and type fields, count of records around ~ 500_000.
So, I need to create an elixir app that fetch this records in batches and pass them (one by one) by REST API interface to other application. Moreover, I need to trackdown progress of entity migration with possible restarts/retries. How I should design this app in OTP terms?
Looks like I need a supervisor with single genserver, that pool data from mongodb in smaller batches (about 100 or more), after that, genserver pushes records via REST API (using Task module), until every record successfully sent and after that proceed next batch, right?
And if I need restarts, when application is stopped or something other happened, should I save every step of batch processing to local db, like ID range of entities and state of batch process, for example progressing / failed / succeded?
And how add to genserver functionality to stop processing? Save genserver status to its state and change status from client code?
If processed entity is duplicated, for REST API it’s OK situation. How should I design this app? Please, share your opinion! Thanks for answers, in advice! 
Most Liked
dimitarvp
Regular stuff: mix new your_app --sup (the CLI option is for it to have an application supervisor).
Not sure you need a GenServer to fetch stuff from Mongo and then send it somewhere else, you can likely use Oban’s free tier and store somewhere in your DB the progress of the sending operation (though not sure if Oban has a connector to Mongo; likely not). But if you really feel you should use a GenServer then yeah, nothing stopping you.
Since you hinted at state being important I’d say yes, these pieces of data should be persisted somewhere. What you proposed as an idea to store ranges can and should work.
There are good articles on how to make your own GenServer but TL;DR you can just send it a message to stop processing and then shutdown itself (if you want that too).
Your question is too broad IMO. Start off with something and start asking a bit narrower questions. 
You will not get it right from the first time and that’s okay.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"