Creating process from GenServer and passing state

[venomnert]

Context:

I am pretty new to OTP. In trying to learn it I’m creating an application that does the following (high level overview):

1. users submit X number tasks to a GenServer.
2. The GenServer iterates over the queue of tasks and creates a process for each task.
3. Once the process is completed it will notify the GenServer it’s done and return its value.
4. Once the GenServer get’s notified it will update the state with the process’s return value, and notify the client with the updated state.

The diagram demonstrates what I’m trying to achieve.

I am able to spawn a process from the GenServer; however, I’m struggling to connect between the GenServer and process, so the process can call the GenServer; in return, the GenServer can handle_call/3.

Questions:

1. Am I taking the right approach of calling a process from a GenServer? Is there better way to create process from GenServer and notify the process about the state?
2. How do I call GenServer from a process and passing the state to server?

[al2o3cr]

This sounds a lot like the Task functionality in the Elixir standard library. Here’s some relevant discussion from 2018, complete with a code example:

Supervising async tasks

I was following this thread due to how common this use case is as expressed by @stefanchrobot and and the solution @josevalim gave seemed interesting and simple. So I went and code it myself as I understood the use case and Task documentation (https://hexdocs.pm/elixir/1.5/Task.html#content). It took me a while to grasp but this is the code. Please correct me if there is anything I misunderstood: Crete the project mix new stask --sup cd stask mix deps.get The supervision tree defmodul…

The tricky part with spawning additional processes is making sure everything works sensibly when things go wrong - what happens when those processes shut down, when the parent shuts down, etc. The Task library wraps the relevant parts (linking, supervising and so on).

[venomnert]

I believe this is what I needed, thank you @al2o3cr! I’m going to give @joaquinalcerro implementation a shot.

[joaquinalcerro]

@venomnert,

Even though this code might work for your current use case you have to consider what happens if one of the tasks crashes.

With this code, if one task crashes, your GenServer will also crash with your state because the Task is still sending the GenServer a message that is currently not handled. You can handle them as follows:

   def handle_info({:EXIT, _task, _reason}, state) do
     IO.puts("Crashed!!!!!")
     {:noreply, state}
   end

  def handle_info({:DOWN, _ref, :process, _pid, reason}, state) do
    IO.inspect(reason)
    {:noreply, state}
  end

The first message will have the tuple {:EXIT, task, reason}. The other one is {:DOWN, _ref, :process, _pid, reason}. You already have a tuple similar to this one but handles the happy path when the task exits normally like this {:DOWN, _ref, :process, _pid, :normal}… note the :normal atom at the end will be pattern matched.

With this in place, your GenServer will not crash and will be available to handle other tasks.

So this might work but: I just wanted to share with you some recommendations @whatyouhide gave us during this weeks ElixirConfLA in Medellin.

He recommended us:

1. Limit the amount of tasks that can be spawned to avoid exhausting the resources

2. Use the proper strategy for your use case:
   a. one_to_one
   b. one_for_all
   c. rest_for_one

   So what happens if one of the task fails? is it worth continue processing the other tasks? how will your state be affected if one task fails?

   Check this link for details of each one: https://hexdocs.pm/elixir/Supervisor.html#module-strategies

3. Nest you supervision tree with other supervisors with the proper strategy. For example, under your main application supervisor, create a worker supervisor with a one_for_one strategy.

4. All processes should be supervised

5. Name your supervisors to access them by name which is easier.

6. Test you supervision tree

I will be monitoring when the talks are published to link it to this thread.

Best regards,

[joaquinalcerro]

You can also check this thread which has a complete example and explanation:

Proper use of Task.async?

As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a large list of similar but different tables. One suggestion I’ve received for dealing with this task was to create a module that would query concretely each table then aggregate them into a single list at the app level vs at the db via unions and view tables. The suggestion was to use Task.async. Since my max concurrent users will never go above 1000 at any g…

Best regards,

[venomnert]

Note

I just wanted to say thank you for taking the time to providing me with extra detailed information.

---

My understanding

I notice the main difference between the handle_info in the other thread and the one you have mention on this thread is the following:

1. Here we are accepting any reasons for :EXIT and :DOWN.
2. In the other post we are accepting a specific case of :EXIT and :DOWN.

Questions

1. However, I’m still unclear as to the reason why we are not pattern matching :normal?

2. In my case is it better to have a single task that handles all of the user submitted tasks or is it better to have a task per user submitted tasks?

Limit the amount of tasks that can be spawned to avoid exhausting the resources

3. Also can you point me towards a resource on testing tasks and supervisor tree?

Test you supervision tree

[joaquinalcerro]

@venomnert,

1. So we are pattern matching the tuples with :EXIT and :DOWN one with the :normal atom and the other one with any reason (you need to pattern match all the cases). The normal case will match when the task finishes successfully without crashing and the other one when it crashes for any reason.

2. As always, depends. If you are having lots of users request to a single GenServer, it might become a bottleneck and you might be better of spawning a GenServer per user request. If that is not the case, you might be already set. Another question: what happens to the state if two users send tasks to the GenServer at the same time? should the state be affected by both user’s tasks? or do you need to have different state for each user request… if this is the case, you might need to spawn a GenServer per user request which will also spawn tasks async.

3. Well Andrea recommended this site: https://github.com/ferd/sups … but he also said it was a bit hard to implement. Having said that, you could use the Observer (in iex session, call it by " :observer.start() ") to see your supervision tree and kill the process to see how is it working.

Hope this helps.