We do something similar and the collection size is also way too big to consider an in memory solution. This happens in the background every couple dozen hours.
Here are few points as to how we approach it:
-
The basic logic is similar to what you describe under continuation-passing style, it is done in a GenServer that is a bit state-machine like (without using fsm) - there is startup routine, crawling, finalizing and waiting states
-
We have a special “cache” DB Table to persist data while it is gathered one page at a time
-
After the crawling is done we “activate” the gathered data by streaming it from the “cache” table to the “live” one (using Repo.stream in transaction), this way the rest of the system always accesses data that was successfully gathered and we never end up with inconsistent state in the live table, any problems would be contained with the “cache” part
-
No circuit breaker but fixed retrying after 5 seconds on failed requests, this part is fully flexible though
-
We have extra status that we persist to ETS table - “waiting”, “crawling”, “retrying”, “finalizing” etc, we also might hold the current error if there is one and the number of page if crawling is in progress, this way UI layer has access to info as to what is happening at the moment
