elnewbee
Design elixir application in OTP terms.
Greetings, everyone! 
When learning elixir, I faced up with the next task.
I have mongodb with stored entities that have id, created_at and type fields, count of records around ~ 500_000.
So, I need to create an elixir app that fetch this records in batches and pass them (one by one) by REST API interface to other application. Moreover, I need to trackdown progress of entity migration with possible restarts/retries. How I should design this app in OTP terms?
Looks like I need a supervisor with single genserver, that pool data from mongodb in smaller batches (about 100 or more), after that, genserver pushes records via REST API (using Task module), until every record successfully sent and after that proceed next batch, right?
And if I need restarts, when application is stopped or something other happened, should I save every step of batch processing to local db, like ID range of entities and state of batch process, for example progressing / failed / succeded?
And how add to genserver functionality to stop processing? Save genserver status to its state and change status from client code?
If processed entity is duplicated, for REST API it’s OK situation. How should I design this app? Please, share your opinion! Thanks for answers, in advice! 
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Bringing learning, work and compliance together since 1987.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First Post!
dimitarvp
Regular stuff:
mix new your_app --sup(the CLI option is for it to have an application supervisor).Not sure you need a
GenServerto fetch stuff from Mongo and then send it somewhere else, you can likely useOban’s free tier and store somewhere in your DB the progress of the sending operation (though not sure ifObanhas a connector to Mongo; likely not). But if you really feel you should use aGenServerthen yeah, nothing stopping you.Since you hinted at state being important I’d say yes, these pieces of data should be persisted somewhere. What you proposed as an idea to store ranges can and should work.
There are good articles on how to make your own
GenServerbut TL;DR you can just send it a message to stop processing and then shutdown itself (if you want that too).Your question is too broad IMO. Start off with something and start asking a bit narrower questions.
You will not get it right from the first time and that’s okay.