Not the person you are replying to, but we also use the same approach.
There are a series of web servers that just serve HTTP traffic, and then there are a series of background servers that just run Oban jobs. The server kind is identified by an environment variable that we check to conditionally start Oban queues or not.
We didn’t need any autoscaling yet so we don’t have any solution for working with that. Anyway, Oban Pro has the DynamicScaler plugin that can do autoscaling for you.
I did not know about that plugin. May be useful sometime where there are jobs that are tied to main db like importing external data to S3 or media processing.
Thanks
This approach is very valid. From financial companies I know that they usually separate their core systems with high availability requirements from other tasks such as background batch jobs, analytics, reporting, and additional services.
To do this you basically have to copy your database and use this copy for everything you don‘t want to run in the core system. Copying the database proves to be a challenging task in itself which needs a dedicated team that does the loading on a daily basis and fixes any load issues that regularly arise. Additionally, the database copy will never be in complete sync with the core database.
@JeyHey you brought up an interesting subject about duplicating the main database or restoring the daily backup to another server and using that for reporting or data extraction tasks. It ca not be used by workers that ingest data into the main database, but for data extraction that does not need to be realtime can be a good solution.
As @dimitarvp said though we need to be careful to not go into all these kind of rabbit holes just for premature optimizations. Only if the metrics tell us that there is something we need to do about it.
For the moment only the duration of a span. Just the fact that you’re starting and then ending a span already automatically gives you a duration. And they can be nested (meaning you can measure the duration of a web response and the DB time that’s part of it, for example). You can also attach all sorts of data to the traces / spans. Pretty awesome stuff.
Dunno if you checked the GitHub repo or their website. Please do and check out the screenshots (they recently added dark mode btw).