Hi,
I wanted to create an application with a supervisor with children processes that read from a file. In order to remember the last read pointer in the file, I would like to keep the file open.
Also, I want to limit the number of children processes so that the supervisor cannot add more than N children. If there is a request to read from the new file, the supervisor must send kill signal to the process that holds the file pointer to least recently read file.
How would I go about doing something like this in Elixir?
Thanks.
1 Like
Not sure what value would you extract from parallel workers all reading from the same file. I/O is usually the #1 bottleneck in 99.9% of apps so not sure how making that even worse is useful.
Can you expand on your use case a bit?
Its separate process per file. There are thousands of unique files to read.
In that case you should use poolboy. I found this article really good at walking you through how:
https://samuelmullen.com/articles/elixir-poolboy-and-littles-law/
Supervisor doesn’t support such elaborate approaches. I believe this can be achieved with Parent.GenServer (full disclosure, I’m the lib author), though it will require some imperative logic to make it work.
The use case is vaguely explained, so I’m not sure this is the best approach. Depending on what you actually want to achieve, perhaps having something like at most N workers doing the reading, and maintaining a separate LRU cache would work better.
DynamicSupervisor supports a max_children option that seems relevant - when there are already the maximum number, start_child will return {:error, :max_children}. Your code could handle that and pick a process to evict before trying again.
Something else to think about: what happens to the state of a file when its process is evicted?
I will give that a shot! Thank you!
