When prefer Oban over AWS Lambda?

Questions / Help
,
1

When do people choose Oban instead of AWS Lambda for async processing?

1 Like
2

Personally, I’d never choose aws lambda.

A serverless approach doesn’t seem to leverage any of the strong points of BEAM, and waiting the startup time of BEAM on such environment is not worth it. So if you’re going serverless, most probably you’re not using elixir.

Oban on the other side can handle running multiple instances of the same service, even if you don’t run your elixir app as a beam cluster.

Just because of that, if your codebase is mainly in elixir, i don’t see any good reasons to use aws lambda.
If you’re dealing with multiple languages or elixir is not your main language, there might be reasons to consider using aws lambda.

edit: and it’s good to keep in mind that oban is not the only way to do background work in elixir. it’s a good one, but there stuff you can handle easily with supervised tasks and other approaches.

9 Likes
3

AWS Lambda is mostly aimed at compiled languages whose binaries start ultra-fast (2ms to 30ms) – think Rust, OCaml, Go, Zig, Nim.

Erlang/Elixir have relatively long startup time – 0.5 to 1.0 seconds – and they are a terrible fit for serverless platforms.

4 Likes
4

Unless one wants to experiment with Firefly :smiley: GitHub - GetFirefly/firefly: An alternative BEAM implementation, designed for WebAssembly

1 Like
5

We have lots of lambdas in python. Startup time is no problem if you keep the lambdas warm.

2 Likes
6

Don’t they get frozen after a bit of inactivity? Maybe that’s what you mean by keeping them warm – you got a background service that pings them every now and then?

1 Like
7

Yeah, you use a timer and ping them every once in a while, it should not be expensive if you plan it well.

That said, AFAIK python has a better startup time than elixir and I agree with you that if you’re using a lot of serverless functions elixir isn’t the best choice.

2 Likes
8

Big thing for me is simplicity. One thing I love with elixir is that you can have pretty much everything in your app + PostgreSQL. I use to be much more into microservices and stuff but in the end you just end up spending so much on connecting everything and development becomes so much more complex.

5 Likes
9

see here: Operating Lambda: Performance optimization – Part 1 | AWS Compute Blog

1 Like
10

Geez, that’s very long. Is it said anywhere exactly how long do they wait until they “freeze” a warmed-up lambda?

1 Like
11

A lot of fine thought here, me personally it’s because:

  1. Simplicity, if you only care about doing basic async processing, elixir / BEAM it self already has tools to do it easily, such as supervised task or process.
  2. For more durable / consistent job, using a dedicated job queue library give better control over retry behavior, uniqueness consideration, worker, etc. over using AWS Lambda + SQS
  3. Avoid vendor lock-in.
  4. Transactional outbox pattern (sending message as part of transaction) could be performed more safely on Oban (rather to SQS directly), because Oban is based on postgres, it could get transactional guarantee of postgres.

That being said, i would probably go with AWS lambda if work i do is:

  1. Pretty complex / hard to do in elixir, enough to justify writing it on another language.
  2. Needing good integration to the rest of AWS ecosystem, such like: Cognito Post-sign up integration, API gateway integration, etc.
3 Likes
12

That depends on available resources in the zone (? I think).
But its at least a couple of minutes, so its possible to keep them warm by pinging them at nearly no cost.
Or if you have enough traffic on the function, thats not needed at all.

1 Like
13

I can surely see the “cost” factor, but to me it just sounds ludacris to regularly ping a service so it doesn’t shut down, when the platform of the service is built to reduce cost by shutting down services when not used – over say a server running full time. So much complexity when one could just use a server, which doesn’t shut down the service in the first place.

6 Likes
14

You’re right, its a little stupid.
But that’s Amazon’s problem, not ours.
AWS allows us to operate a complex distributed system with close to zero dev-ops cost. Moving the lambdas to EC2 would drastically increase the costs. I know because that is how it was before. We could also have transformed the old system to Elixir, which would also have reduced complexity, but:

(also I did not know about the goodness of Elxir/OTP then)

2 Likes
15

There are some great points in this thread. Here are a few other things to consider on behalf of Oban compared to a stateless background job systems (and even some other persistent ones):

  1. Observability — built in metrics for queue time and execution time, as well as durable timestamps that identify exactly when a job was completed.
  2. Error Handling — built in retries, automatic backoff, and persisted error messages along with stacktraces.
  3. Concurrency Controls — concurrency per-queue, because infinite scalability isn’t always the right behaviour. What about external rates limits, or database contention?
  4. Testing — you run jobs for side effects, and you typically want to test the complete flow from when a job is enqueued to its execution. Testing execution flows should be simple to control without heavy mocking.
5 Likes
16

After reading through the responses, it sounds like elixir is not a first class citizen (yet?) within AWS lambda, and so it’s not a realistic option without contortions.

I guess my core interest is when to use a self administered, featureful job scheduler like Oban versus externally administered/managed AWS lambda, in a world where there is no penalty or immediate issues with using Elixir in a lambda. That’s too hypothetical for now, so I will revisit in a year or so, if there is more first party or third party support for Elixir in a lambda.