Only create process after task finishes (ConsumerSupervisors + db inserts)

Questions / Help
1

Hi all!

I’m currently using ConsumerSupervisors to launch 1 task per event but the consumer currently makes a direct insert to the db causing serialization error. I even put in a sleep timer in the consumer to not launch a new task giving the old task time to complete but I still get the error.

Is there a way to tell the supervisor not to launch a new task till it finishes the current one? I can only see which_children in the doc but I think it might be taking it to the extreme to check the children and their status.

Thanks in advance! Cheers!

possible solutions:

  • task aync/await?
2

ConsumerSupervisor:

The minimum amount of concurrent children per producer is specified by :min_demand and the maximum is given by :max_demand .

Given your question, you seem to want to configure both :min_demand and :max_demand to 1 - though that seems to defeat the intent behind ConsumerSupervisor - so it is entirely possible that I’m misunderstanding your question.

3

Thanks for the reply!
I was wondering if there is another way to somehow bypass the db transaction error and make it still concurrent. =( I was wondering if I should remove the statements from the transaction but I think it might still cause an error. :confused: I had to do a transaction in order to re-create an upsert for redshift. (stage data -> delete from source -> insert from stage data)