Separating Workers From Apps?

Phoenix Forum Questions / Help
1

I currently have a phoenix app, and have introduced workers into it. Currently the workers are starting along with the application using mix phx.server. But this is unwanted - I don’t want to spin up workers and apps together. I want to start apps to serve traffic, and start worker processes to work.

I’m used to having app and worker processes separated and scaling them separately. As such, I might have one app server on one box and 3 workers on other boxes.

How would I set that up through Phoenix? Essentially, I want to load my app code into the workers without starting servers or loading any other unnecessary libraries. Appreciate any pointers.

2

If you want to start things independently then they should be different applications. You could use an umbrella project structure to to still have the conveniences of starting things as a single unit in development though. This is also in no way specific to phoenix at all.

2 Likes
3

Thanks for the suggestion - it is related to Phoenix in that I was looking how I might access a Phoenix application, along with models and schema definitions and business logic from my worker processes. I’m not going to duplicate that code into another application, and apart from loading the application code into the worker process, only other option I can see is to create libraries shared across two separate applications.

4

Thanks for the suggestion - it is related to Phoenix in that I was looking how I might access a Phoenix application, along with models and schema definitions and business logic from my worker processes.

In an app with workers it might be a good idea to separate “data layer” (models and schemas) into its own otp app. And have phoenix otp app and worker otp app both make it their dependency.

I think it’s not related to phoenix in a sense that phoenix doesn’t actually manage any of your data – ecto does.

1 Like
5

Yeh, this sounds more like what I want to do long-term! Much cleaner.

You’re right - what I’m trying to do is run my app twice for different purposes. Not great. Still trying to shake bad habits from Rails/Sidekiq :confused:

Only reason I related it to Phoenix is because I was hoping there was a way to pass in an environment variable to determine which supervisors to run and, as such, whether to fire up a worker or server endpoint (looking at application.ex). But that’s quite messy. I think I’ll go with a shared app.

6

You can do it with MIX_ENV, but it would only be helpful during compilation. Like this

defmodule PhoenixApp.Application do
  use Application

  def start(_type, _args) do
    import Supervisor.Spec

    children = case unquote(Mix.env()) do
      env when env in [:dev, :test] -> [supervisor(PhoenixApp.Endpoint, []), worker()]
      :prod -> [supervisor(PhoenixApp.Endpoint, [])]
    end

    opts = [strategy: :one_for_one, name: PhoenixApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end
7

That’s fine. I can compile the two different versions (app, worker). It’s still not ideal, but it’s even simpler than breaking out another application just to share dependencies between them and then having to manage those dependencies when I roll updates.

8

It’s still not ideal, but it’s even simpler than breaking out another application just to share dependencies between them and then having to manage those dependencies when I roll updates.

I actually find separated apps much simpler to write. In any non-trivial web related project that I start now, first thing I do is, I separate data and web layers. Never had problems with updates either.

9

Sounds like a good habit, to be fair. Still coming at things from an MVC perspective…

So if you separate out the data layer, what is responsible for validations related to your web layer? Do you pipe data through validation modules in your web layer and then pass them onto your shared data layer code?

10

What kind of validations?

I try to keep my web layers as dumb as possible. They usually just accept data and pass it to domain layer. domain layer then contains all business logic. If it’s not complicated, domain and data can be in the same otp app.

1 Like
11

Based on user input, for example. Typically, we would have a changeset in the model that pipes values through validation logic. But those validations would only make sense on the web layer, not the worker.

12

Changesets and any other ecto-related functionality go into data otp app.

If workers don’t belong with data layer, so just like web, they shouldn’t care about those validations. They usually go into their own otp app as well.

13

Ok, I think I’m starting to come around to this separation. I like.

So I have:

web - controllers, views, dumb
domain and data - business logic, models, etc
worker

1 Like
14

There’s no need to bake in the branch at compile time, you can do a runtime check like if System.get_env("USE_PHOENIX"), do: ...

2 Likes
15

I think I might start with this at first. I’m using Docker, so I can just pass in the relevant ENV variable when running different instances of the same image.

16

I mentally devide two kinds of validtion. Input validation in the weblayer (a text form must send text) and domain validation in the domain layer (e.g. a table reservation date must be at least two weeks in advance). The latter is certainly as important to your worker as it is to your weblayer.

3 Likes
17

Decided to run with the ENV variable check and it’s working quite well…

if System.get_env("APP_WORKER") do
  children = [
    supervisor(Verk.Supervisor, [])
  ]
else
  redis_url = Confex.get_env(:verk, :redis_url)

  # Define workers and child supervisors to be supervised
  children = [
    # Start the Ecto repository
    supervisor(Pink.Repo, []),
    # Start the endpoint when the application starts
    supervisor(PinkWeb.Endpoint, []),
    # Start your own worker by calling: Pink.Worker.start_link(arg1, arg2, arg3)
    # worker(Pink.Worker, [arg1, arg2, arg3]),
    worker(Redix, [redis_url, [name: Verk.Redis]], id: Verk.Redis)
  ]
end

This allows me to do:

mix phx.server -> run app
APP_WORKER=true mix phx.server -> run worker

Of course, there are drawbacks, but this is a pattern that works well for many production apps, so I’m happy to run with it for now until I need to separate out the code into different layers and services.

18

It’s usually suggested to assign values outside of if (see “Deprecation of imperative assignment”)

children = if System.get_env("APP_WORKER") do
  [
    supervisor(Verk.Supervisor, [])
  ]
else
  redis_url = Confex.get_env(:verk, :redis_url)

  # Define workers and child supervisors to be supervised
  [
    # Start the Ecto repository
    supervisor(Pink.Repo, []),
    # Start the endpoint when the application starts
    supervisor(PinkWeb.Endpoint, []),
    # Start your own worker by calling: Pink.Worker.start_link(arg1, arg2, arg3)
    # worker(Pink.Worker, [arg1, arg2, arg3]),
    worker(Redix, [redis_url, [name: Verk.Redis]], id: Verk.Redis)
  ]
end
2 Likes
19

Was mainly because I don’t need to bind redis_url unless I am using Redix… but, having said that, I don’t actually need to do any of it - I can just use Confex directly or even fetch config from my App module.

Thanks for the link! I’ll take that into account in future.

…though doesn’t look like that advice is relevant to my code. I’m binding once and it is only being used within that branch, so is not, as per the link “implicitly changing the value”.

20

It’s relevant because your code conveys the impression that you haven’t internalized that Elixir is expression based rather than statement based. Your style uses if like a statement, which if isn’t; both the if and the else still return a value - it simply isn’t bound to anything outside of the if expression. if is a macro but works like the ternary operator (cond ? true : false) in other languages - it simply returns nil when you don’t hand it an else expression.

1 Like