elnewbee
Greetings, everyone! 
When learning elixir, I faced up with the next task.
I have mongodb with stored entities that have id, created_at and type fields, count of records around ~ 500_000.
So, I need to create an elixir app that fetch this records in batches and pass them (one by one) by REST API interface to other application. Moreover, I need to trackdown progress of entity migration with possible restarts/retries. How I should design this app in OTP terms?
Looks like I need a supervisor with single genserver, that pool data from mongodb in smaller batches (about 100 or more), after that, genserver pushes records via REST API (using Task module), until every record successfully sent and after that proceed next batch, right?
And if I need restarts, when application is stopped or something other happened, should I save every step of batch processing to local db, like ID range of entities and state of batch process, for example progressing / failed / succeded?
And how add to genserver functionality to stop processing? Save genserver status to its state and change status from client code?
If processed entity is duplicated, for REST API it’s OK situation. How should I design this app? Please, share your opinion! Thanks for answers, in advice! 
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #hex
- #security
Our Sponsors
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 1- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
dimitarvp
Regular stuff:
mix new your_app --sup(the CLI option is for it to have an application supervisor).Not sure you need a
GenServerto fetch stuff from Mongo and then send it somewhere else, you can likely useOban’s free tier and store somewhere in your DB the progress of the sending operation (though not sure ifObanhas a connector to Mongo; likely not). But if you really feel you should use aGenServerthen yeah, nothing stopping you.Since you hinted at state being important I’d say yes, these pieces of data should be persisted somewhere. What you proposed as an idea to store ranges can and should work.
There are good articles on how to make your own
GenServerbut TL;DR you can just send it a message to stop processing and then shutdown itself (if you want that too).Your question is too broad IMO. Start off with something and start asking a bit narrower questions.
You will not get it right from the first time and that’s okay.