merongivian

merongivian

Task.Supervisor stops after one child fails on restarts

I have a task supervisor configured through a phoenix application, it will restart a failing child twice if it fails:

lib/nanoindie/application.ex

# children configuration on start
    children = [
      supervisor(Nanoindie.Repo, []),
      supervisor(NanoindieWeb.Endpoint, []),
      supervisor(Task.Supervisor, [[name: Nanoindie.TaskSupervisor, max_restarts: 2]]) # task supervisor
    ]

I add the children dynamically through start_link, what i do here is retrieving some info from blogs via web crawling

lib/nanoindie/jobs/blogs.exs

Enum.map Repo.all(Blog), fn(blog) ->
  Task.Supervisor.start_child(Nanoindie.TaskSupervisor, fn ->
      # web crawling
    end
  end, restart: :transient)
end

When i execute this, and for some reason one of the child fails, it will restart it normally, but if all restarts of this child fail it won’t run the rest of the processes

if i use restart: :temporary instead of restart: :transient it works fine, all processes run normally (without restarts)

Is this the right behavior?, i was reading through the DynamicSupervisor’s code and it seems that it will stop the dynamic supervisor if one the children doesn’t have any successful retry, i might be wrong though, im still learning elixir and OTP :slight_smile:

Most Liked

svilen

svilen

Author of Concurrent Data Processing in Elixir

The solution for me was to put a Supervisor in front of each job; so we ended up with 1 Dynamic Supervisor, that would start Supervisors with restart: temporary, and let the Supervisor manage the GenServer process with restart: :transient. A failing job would exit, bring down its Supervisor, which will be just killed by the Dynamic Supervisor above, leaving all other running jobs and their supervisors intact.

mbuhot

mbuhot

I recently saw Task.Supervisor suggested as a simple alternative to a sidekiq style job queue in mistakes-rails-developers-make-in-phoenix-pt-1-background-jobs

Given the way a single failing job will take out all other in-progress jobs after several failures, I’m wondering if this is actually an anti-pattern?

Where Next?

Popular in Questions Top

chokchit
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2733ms. You can configure how long re...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New

Other popular topics Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 39467 209
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36352 110
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement