Benjamin-Philip

Benjamin-Philip

Run tasks with a delay

Suppose I have the following function:

def foo, do: IO.puts "Foobar"

I want to run it a minute after some work done in my function:

def fun(x) do
  somework(x)
  # Schedule foo to run after 1 minute
end

I do not want to delay the return of fun by 1 minute - I want to return immediately after somework returns (Otherwise I can just do Process.sleep). I also do not care if foo fails or not.

One thing I have done is the following:

Task.start(fn ->
  Process.sleep(60_000)
  &foo
end)

But if I am not mistaken, another process is started to run the task. Having a process sitting idle for a minute seems wasteful of system resources - especially if the delay is much greater than a minute and there are many tasks being created per second.

Another idea I got is to write a GenServer to handle scheduling tasks, but I avoided this because this seemed like a common pattern, and expected there to be a “preferred” solution. Also, this sort of functionality is probably better off in Task.Supervisor than a hand-written GenServer anyway.

How can I achieve this using the existing task running infrastructure?

First Post!

soup

soup

It may be useful to know more details about your use-case specifically.

I wouldn’t worry about having too many sleeping processes. The BEAM can manage many thousands of processes as they are lighter weight than a typical OS process.

You can also use GenServer as you said, probably mostly depends how many tasks your running and the structure of them. If you are going to be booting many a second, while also not caring about success, I might use a GenServer simply to act as a throttler/circuit break later down the line, or to distribute it.

Depending on the context, you could use send_after too, if the owning process will still be around to receive the message. (You would probably use send_after in your GenServer, but I mean you could also use it in a LiveView for example (which are just GenServers anyway).)

There is a great talk by Sasa Juric that talks about BEAM and processes that might provide a bit of insight to the model, though it doesn’t answer your question specifically https://www.youtube.com/watch?v=JvBT4XBdoUE.

Most Liked

al2o3cr

al2o3cr

What should happen with records that are left behind in the database - for instance, if the BEAM is restarted after one is inserted but before the deletion?

The answer to that may suggest alternative approaches:

  • if leftover records should be periodically cleaned up, does there need to be a per-record deletion process at all?
  • if leftover records are ignored, do any of them need to be deleted?

Another thing to think about: while @LostKobrakai is 100% correct that sleeping processes are basically free, having an unlimited number of them wake up simultaneously can still create a mess. Here’s a scenario:

  • a fast API client calls the endpoint that inserts records + schedules deletion 1000 times in 5s. Easy to do, and the server has no problem handling the requests
  • X minutes later, 1000 processes all wake up, and all try to run a Repo.delete.
    • some of them will manage to grab a connection and send a query to the DB
    • then the connection pool will run out. Now every connection in the pool is running a Repo.delete statement.
    • everything else starves: the other 900-some waking-up processes, and every Phoenix request handler block waiting for a DB connection
    • if the Repo.delete statements take too long (~100ms or so with default settings) then waiting processes will start to crash with messages like: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after NNNNms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information

The result is that a sufficiently-large-and-fast burst of requests to the API creates a brief whole-server outage a few minutes later.

Solutions like a job queue or a worker pool avoid this “thundering herd” problem by only trying to do a fixed amount of work at once.

dimitarvp

dimitarvp

Think we can safely count this thread in the XY problem category? :smiley:

(We don’t have one but maybe we should? :003:)

Still, to answer the original question, your best bet for a periodic worker would be to just register a worker inside your supervision tree and have it recurse infinitely:

# lib/your_app/periodic_delete_worker.ex
defmodule YourApp.PeriodicDeleteWorker do
  use Task, restart: :permanent

  def start_link(_arg) do
    Task.start_link(__MODULE__, :run, [])
  end

  def run do
    # Do your stuff here

    # Wait for a message for 1 minute.
    # If no message is received, still wake up after the 1 minute has passed
    # and reschedule the same worker.
    receive do
    after
      60_000 -> run()
    end
  end
end

And then:

# lib/your_app.ex
defmodule YourApp do
  use Application

  def start(_type, _args) do
    children =
      [
        YourApp.Repo, # just an example!
        YourApp.PeriodicDeleteWorker # 👈 this is what you want
      ]

    options = [strategy: :one_for_one, name: YourApp.Supervisor]

  Supervisor.start_link(children, options)
end
Benjamin-Philip

Benjamin-Philip

For reference, I finally went with a periodic worker. Turns out, talking to the person reviewing your PR before posting on the internet is a good idea!

Where Next?

Popular in Questions Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

Other popular topics Top

minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 54996 245
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

We're in Beta

About us Mission Statement