Hello all,
I’ve configured the following setup in application.ex
:
{FLAME.Pool,
name: Nexus.HTMLToPDFRunner,
min: 1,
max: 10,
max_concurrency: 5,
idle_shutdown_after: 10_000,
log: :debug}
But, my application is deployed across multiple instances (fly scale count 3
). As a result, I end up with 3 instances of the flame runner instead of the single one I intended to have.
Does anyone have insights on how to configure this so that I only get one flame runner, regardless of the number of deployed instances?
Hey @vladnec, I don’t have an answer to your question, but I think some more information might be useful.
Specifically, are your multiple instances clustered or do they just share a DB? Without clustering, I’m not sure how Flame on one instance would know that another instance has a runner.
hei @NduatiK,
my instances are clustered - this is part of my application.ex
configuration
def start(_type, _args) do
flame_parent = FLAME.Parent.get()
children = children_process_list(flame_parent != nil)
opts = [strategy: :one_for_one, name: Nexus.Supervisor]
Supervisor.start_link(children, opts)
end
def children_process_list(false) do
[
....
# setup for clustering
{DNSCluster, query: Application.get_env(:nexus, :dns_cluster_query) || :ignore},
...
]
end
def children_process_list(true) do
[
....
# DNS Cluster is not configured here, when the runner is a flame runner.
...
]
end
Hey, I had some time to read some of the FLAME code, and it looks like Flame.Pool
only considers the min value on a single node when starting up.
I have asked the same question on the fly forum and they mentioned that it is made by design like that (to have multiple Flame runners for each node )
So, are you sure about it?
I mean, i was able to prove that for each node i spawn, i get a new flame runner.
You can totally do this. You can have a global singleton which is your FLAME pool. For example, using :pogo
:
-
Remove FLAME.Pool
from your application children.
-
Add Pogo supervisor to your application children (check the pogo docs). Let’s say you called it MyApp.DistributedSueprvisor
-
Start your global singleton FLAME pool supervisor like this:
Pogo.DynamicSupervisor.start_child(
MyApp.DistributedSupervisor,
FLAME.Pool.child_spec(
name: MyApp.FlamePool,
min: 20,
max: 30,
max_concurrency: 10,
idle_shutdown_after: 2 * 60 * 60 * 1_000,
min_idle_shutdown_after: 2 * 60 * 60 * 1_000,
timeout: 5_000
)
)
Pogo will ensure unique children IDs. Flame will set the ID to be some kind of tuple like {FLAME.Pool, MyApp.FlamePool}
(it uses the name
opt). So if you are using the pogo supervisor which_children
function, watch out for that.
Do not copy my Flame config! Decide your own values. This is just a concept BTW. I am not recommending you use a single flame supervisor on just one node. It seems at odds with the spirit of FLAME.
1 Like
I agree with you, I meant that the code shows that for each instance the number of min workers is managed locally, without considering whether workers exist on other instances.