I’m having a little difficulty understanding where GenServer will fit into my planned architecture. I’m used to using background workers (Sidekiq in Ruby) along with persistent stores (Redis), retries, scheduling, etc, etc, and also queuing tech such as RabbitMQ.
From what I read after searching, GenServer is a great for firing off lightweight background processes. But then I found this:
" If something goes wrong, you will not get notified back that processing of message you sent using cast failed. For example, you won’t know that sending email failed since you don’t wait for a reply"
Understandable. But lets say I need these jobs to reprocess if they fail, or need to attach other such logic, is GenServer capable of that, or is the use-case a little more limited? I’ve also been reading into supervisors and fault tolerance, and came upon this:
“In Elixir, supervisors are tasked with restarting processes when they fail. Instead of trying to handle all possible exceptions within a process, the “Let it crash”-philosophy shifts the burden of recovering from such failures to the process’ supervisor.”
Let’s say I have a scraper job that may use an API which rotates proxies (Crawlera, for eg) that can either return a success or too many requests/banned response. Usually, I would have the job re-queue if not successful, and continue to retry, perhaps increasing the increments between retries. Is GenServer and supervisors a good use-case for this kind of logic, or do I need to look into some other solution?
This suggests it might be a good for my use-case, and would be a healthy introduction into OTP:
While this suggests not so, and that perhaps exq, toniq or verk would be a better solution:
Basically, do I have all the tools I need to build durable jobs, retries with exponential backoffs, dynamically scheduled jobs with GenServer/Supervisors/Mnesia?