Task.async works with Cowboy but not with Bandit

Questions / Help
,
1

I have the following situation. I have an umbrella app, with a few “subapps”. One of those is an admin frontend from where i can trigger an import job. This import job starts an import of an xml file into the database. This works fine when the admin app uses cowboy but not when i switch to Bandit.

So, when i click the “Import” button in the frontend, this triggers an action in the controller and there i call the import function, which comes from another app, in a Task.async.

def import(conn, %{"key" => key}) do
    Task.async(fn ->
      try do
        App2.Importer.import_from_s3(key)
      rescue
        ex ->
          error_msg = Exception.format(:error, ex, __STACKTRACE__)
          Logger.error("[Admin] Error durring import/export: #{error_msg}")
      end
    end)

    conn
    |> put_flash(
      :info,
      "Started Import in the background (go to /logs to check the progress)"
    )
    |> redirect(to: ~p"/import-jobs")
end

When i use cowboy, this works fine locally and in production. I am running the app in production in a docker container. Locally not, there I have Erlang and Elixir installed via asdf.

When I use Bandit, this does not work, wether locally nor in production. Basically the only thing I change is in the config of the Endpoint, to set the adapter to Bandit.PhoenixAdapter. I use Bandit in the newest version 1.6.2.
In development (locally) the App2.Importer.import_from_s3(key) starts to run, i see the first logs, but it cancels after a few seconds. No Errors, no logs.
In produciton the App2.Importer.import_from_s3(key) is not running at all. No Errors, no logs, nothing.

Erlang: 27.2
Elixir: 1.18.1
OS: debian-bullseye-20241223-slim

Has anybody any idea what that could be? I cannot really give more code, as this is a production app from the company.

2

The behaviour around when request handing processes are shut down might be different between bandit and cowboy. If you want your import to finish async after having send a response and finished handling the request I’d suggest making sure it’s not linked or otherwise coupled with the request process.

1 Like
3

Yeah I think it’s different. From what I remember, Cowboy maintains a pool of workers, and after a request has been handled successfully, the process is not scraped but returned back to the pool of workers. If it crashes, a new process is created, but the default is that it stays alive, and handles next request.

Bandit I think is shutting down request workers even after they completed successfully, which has its advantages for sure avoiding unintended situations related to linking, or process dictionary etc. like the above.

I think the behavior where the presented code works is actually wrong. It works by accident, because the request process does not die.

The proper way of handling the situation would be not starting the Task as linked process to the web request, but instead use TaskSupervisor as described here:

https://hexdocs.pm/elixir/1.12/Task.html#module-dynamically-supervised-tasks

I also think @einstein that you should await on your Task, always, if you’re starting with Task.async. You probably want to use here Task.start or Task.start_link. If you use Task.start you don’t technically have to use TaskSupervisor.

3 Likes
4

Afaik this is even further different between http1 and http2, so one really shouldn’t depend on those process lifecycles.

2 Likes
5

This can be, I don’t know but seems likely.

6

Thanks @hubertlepicki and @LostKobrakai for the insights. @hubertlepicki your suggestion to use Task.start solved the issue. That is want I wanted, I dont want to await or start a link or anything, I just want to start that process in the background.

7

As the others said, your code doing Task.async working was an accident – don’t do that. I’d also say don’t do Task.start either, you are better off supervising all such background processes.

I get how someone can sometimes be annoyed and just want the problem to go away but you should do things the right way, meaning it should be stable long-term, not to mention observable. Starting random tasks in the background is neither of those things.

3 Likes
8

To be absolutely honest, probably the best solution would be something like Oban here.

3 Likes