Benjamin-Philip

Benjamin-Philip

Run tasks with a delay

Suppose I have the following function:

def foo, do: IO.puts "Foobar"

I want to run it a minute after some work done in my function:

def fun(x) do
  somework(x)
  # Schedule foo to run after 1 minute
end

I do not want to delay the return of fun by 1 minute - I want to return immediately after somework returns (Otherwise I can just do Process.sleep). I also do not care if foo fails or not.

One thing I have done is the following:

Task.start(fn ->
  Process.sleep(60_000)
  &foo
end)

But if I am not mistaken, another process is started to run the task. Having a process sitting idle for a minute seems wasteful of system resources - especially if the delay is much greater than a minute and there are many tasks being created per second.

Another idea I got is to write a GenServer to handle scheduling tasks, but I avoided this because this seemed like a common pattern, and expected there to be a “preferred” solution. Also, this sort of functionality is probably better off in Task.Supervisor than a hand-written GenServer anyway.

How can I achieve this using the existing task running infrastructure?

First Post!

soup

soup

It may be useful to know more details about your use-case specifically.

I wouldn’t worry about having too many sleeping processes. The BEAM can manage many thousands of processes as they are lighter weight than a typical OS process.

You can also use GenServer as you said, probably mostly depends how many tasks your running and the structure of them. If you are going to be booting many a second, while also not caring about success, I might use a GenServer simply to act as a throttler/circuit break later down the line, or to distribute it.

Depending on the context, you could use send_after too, if the owning process will still be around to receive the message. (You would probably use send_after in your GenServer, but I mean you could also use it in a LiveView for example (which are just GenServers anyway).)

There is a great talk by Sasa Juric that talks about BEAM and processes that might provide a bit of insight to the model, though it doesn’t answer your question specifically https://www.youtube.com/watch?v=JvBT4XBdoUE.

Most Liked

al2o3cr

al2o3cr

What should happen with records that are left behind in the database - for instance, if the BEAM is restarted after one is inserted but before the deletion?

The answer to that may suggest alternative approaches:

  • if leftover records should be periodically cleaned up, does there need to be a per-record deletion process at all?
  • if leftover records are ignored, do any of them need to be deleted?

Another thing to think about: while @LostKobrakai is 100% correct that sleeping processes are basically free, having an unlimited number of them wake up simultaneously can still create a mess. Here’s a scenario:

  • a fast API client calls the endpoint that inserts records + schedules deletion 1000 times in 5s. Easy to do, and the server has no problem handling the requests
  • X minutes later, 1000 processes all wake up, and all try to run a Repo.delete.
    • some of them will manage to grab a connection and send a query to the DB
    • then the connection pool will run out. Now every connection in the pool is running a Repo.delete statement.
    • everything else starves: the other 900-some waking-up processes, and every Phoenix request handler block waiting for a DB connection
    • if the Repo.delete statements take too long (~100ms or so with default settings) then waiting processes will start to crash with messages like: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after NNNNms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information

The result is that a sufficiently-large-and-fast burst of requests to the API creates a brief whole-server outage a few minutes later.

Solutions like a job queue or a worker pool avoid this “thundering herd” problem by only trying to do a fixed amount of work at once.

dimitarvp

dimitarvp

Think we can safely count this thread in the XY problem category? :smiley:

(We don’t have one but maybe we should? :003:)

Still, to answer the original question, your best bet for a periodic worker would be to just register a worker inside your supervision tree and have it recurse infinitely:

# lib/your_app/periodic_delete_worker.ex
defmodule YourApp.PeriodicDeleteWorker do
  use Task, restart: :permanent

  def start_link(_arg) do
    Task.start_link(__MODULE__, :run, [])
  end

  def run do
    # Do your stuff here

    # Wait for a message for 1 minute.
    # If no message is received, still wake up after the 1 minute has passed
    # and reschedule the same worker.
    receive do
    after
      60_000 -> run()
    end
  end
end

And then:

# lib/your_app.ex
defmodule YourApp do
  use Application

  def start(_type, _args) do
    children =
      [
        YourApp.Repo, # just an example!
        YourApp.PeriodicDeleteWorker # 👈 this is what you want
      ]

    options = [strategy: :one_for_one, name: YourApp.Supervisor]

  Supervisor.start_link(children, options)
end
Benjamin-Philip

Benjamin-Philip

For reference, I finally went with a periodic worker. Turns out, talking to the person reviewing your PR before posting on the internet is a good idea!

Where Next?

Popular in Questions Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New

Other popular topics Top

nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement