Task.Supervisor.start_child can be a memory problem?

[shahryarjb]

@al2o3cr

that match_delete is going to need to examine every entry to do its job. Consider a secondary index to avoid this, as in a relational database

I consider a secondary strategy for the time if a problem happens and ram is cleaned; A Genserver Get all refresh token in handle_continue

@impl true
def handle_continue(:sync_with_database, state) do
  UserToken.revaluation_user_token_as_stream(&save(
    %{ ... }
    },
    &1.user_id
  ), %{expire_time: DateTime.utc_now()})
  {:noreply, state}
end

def save(user_token, user_id) do
  ETS.Set.put!(table(), {String.to_atom(user_token.token_info.token_id), user_id, user_token.token_info})
end

def revaluation_user_token_as_stream(action) do
  from(t in UserToken)
  |> MishkaDatabase.Repo.stream()
  |> run_action_repo_stream(action)
end

defp run_action_repo_stream(stream, action) do
  MishkaDatabase.Repo.transaction(fn() ->
    stream
    |> Task.async_stream(action, max_concurrency: 10)
    |> Stream.run
  end)
end

I created a query which is stream in Ecto side and call save function in each Task.async_stream to send refresh tokes to ETS.

But the access tokens are going to be deleted, and it is not important because user can create it with refresh token

@al2o3cr

If UserToken.revaluation_user_token_as_stream does significant computation / IO / etc, your system is likely to run out of those resources before a single Supervisor managing their distribution is a major problem.

I just do 2 jobs, the first one is getting data or deleting a record from Postgres with Ecto that you can see in this comment I put in top as revaluation_user_token_as_stream function

The second one is rejecting the tokens are expired with this function:

def delete_expire_token() do
    time = DateTime.utc_now() |> DateTime.to_unix()
    pattern = [{{:"$1", :"$2", :"$3"}, [{:<, {:map_get, :access_expires_in, :"$3"}, time}],[true]}]
    ETS.Set.select_delete(table(), pattern)
  end

I need to let users load and delete a record from ram not database but in background I want to sync them.

I think with my previous explanations in this post if you are in my place and are creating an Open source CMS for a normal scale what I example in this post what do you do?

We have some limitations. First, we do not want to use external software except Postgrass.
Second, we are going to cover the high scale of the user and be ready for distributing in the future.
Thirdly, it makes me a good learner and also a good software for people who want to test, otherwise they can use Joomla or WordPress.

---

I really want to edit my MishkaCms entire structure over time and provide a good service to users in the future versions that are productions.

It should be noted, I can not create a queue to response my user about their tokens, it should be accessible every time and very fast as possible

---

@al2o3cr

If UserToken.revaluation_user_token_as_stream doesn’t do significant computation / IO / etc, it doesn’t belong in a Task at all.

Some activities in this system should not keep the user waiting and should be done in the background and their output is not important but should be done quickly. They should not be in the queue, and it is very important that they are applied quickly.

Thank you in advance

---

Moving to ETS based on:

1.Optimizing Your Elixir and Phoenix projects with ETS - DockYard
2. Genserver VS ETS - #10 by benwilson512
3. Help to understand PartitionSupervisor in elixir 1.14