DynamicSupervisor sharding helpful?

Questions / Help
1

I’ve got a DynamicSupervisor that seems to crash when spinning up tens of thousands of child processes sometimes.

I’m still in the process of trying to figure out the problem… but was wondering:

In an earlier proposal, https://github.com/elixir-lang/gen_stage/issues/10, DynamicSupervisor apparently has the concept of shards – but this was later changed it seems to be based on GenStage behaviors?

Would sharding the DynamicSupervisor help me deal with bursts of process creation or other issues?
Or should I work more at rate-limiting things?

Thanks,
Scott S.

1 Like
2

It could be slow but it shouldn’t randomly crash… I don’t think we can tell if it would help without knowing what the issue is.

What’s the crash message?

1 Like
3

I take it you mean partitioning?

In any case - yes, partitioning is a way to distribute load, but I do think you should never haul out the big guns before you know exactly what is causing the “seems to crash” behaviour. At the end of the day, a DynamicSupervisor is just a GenServer, (IIRC) a start_child is just a GenServer call, so a rapid influx of these messages would at best just fill up the mailbox while your supervisor patientily processes the messages in it.

1 Like
4

Yes, I guess partitioning would be the correct then, thanks! :slight_smile:

And you’re definitely right about finding the root cause – it appears that we had a silent failure elsewhere that lead to processes terminating prematurely!

5

I was trying to use the same terminology that José Valim used in his original proposal:

Proposal for DynamicSupervisor · Issue #10 · elixir-lang/gen_stage · GitHub

The DynamicSupervisor is going to provide automatic sharding. Imagine
the following start_link call:

DynamicSupervisor.start_link(MySupervisor, args, )
it will start a single supervisor with the specification defined by
MySupervisor. By passing the :shards option, the DynamicSupervisor
will start N supervisors (let’s call them shards) under the parent
supervisor with the specification defined by MySupervisor:

6

Yeah, these young kids, always sloppy with words :wink:

(I still remember when someone used “sharding” the first time and I was all confused and had to go on the internet to find out that all I missed was someone coining a new fashion term for existing techniques; given that our industry is 99% fashion-driven this term seems to have stuck)

2 Likes
7

LOL! I thought the same about sharding in regards to partitioning when I found out what it was… ^.^;
I don’t get the whole thing with renaming things either… ^.^;