Basic: Missing a fundamental concept with applications, appreciate help - `mix run` not waiting

Questions / Help
1

I’m trying to setup a worker for my application, so I’ve setup a simple application separately.

I’m using Verk, and from their docs:

defmodule Example.App do
  use Application

  def start(_type, _args) do
    import Supervisor.Spec
    tree = [supervisor(Verk.Supervisor, [])]
    opts = [name: Simple.Sup, strategy: :one_for_one]
    Supervisor.start_link(tree, opts)
  end
end

However, when I hit mix run, it compiles and runs the application, and yet immediately exits. I would expect it to stay up as the supervisors run. I suspect I am missing something else which will keep the application running/waiting.

Similar to this:

Except I’m not using distillery yet - just want mix run to fire up my app in dev and have that app waiting.

Would appreciate some pointers. I know this is basic stuff, so play nice please.

I have recently found the --no-halt flag - but shouldn’t this logic be inside my application?

1 Like
2

Can you post mix.exs for this app?

1 Like
3

Pretty standard, really:

defmodule MyApp.Mixfile do
  use Mix.Project

  def project do
    [
      app: :myapp,
      version: "0.0.1",
      elixir: "~> 1.4",
      start_permanent: Mix.env == :prod,
      deps: deps()
    ]
  end

  def application do
    [
      mod: {Example.App, []},
      extra_applications: [:logger]
    ]
  end

  defp deps do
    [
      {:verk, "~> 1.0"},
      {:distillery, "~> 1.5", runtime: false}
    ]
  end
end

mix run without --no-halt:

[info] Schedule Manager started
[info] Queue Manager started for queue default
[info] Workers Manager started for queue default (running)
[info] Added 0 jobs.
[info] No more jobs to be added to the queue default from inprogress list.
[info] Queue Manager started for queue priority
[info] Workers Manager started for queue priority (running)
[info] Added 0 jobs.
[info] No more jobs to be added to the queue priority from inprogress list.

…then exits immediately. As if I am running it as a script.

As I say, I think I am missing some fundamental concept behind how applications work.

1 Like
4

See this for a starting point. If my understanding is correct mix essentially starts the processes in it’s own OS process and after it finishes the starting script it exits - taking the BEAM with it. In a development environment you are meant to use iex -s mix to run the application (vs. running a script).

5

Thanks. I did find that before, and was a little confused with:

def start(_type, _args) do
  IO.puts "starting"
  Task.start(fn -> :timer.sleep(1000); IO.puts("done sleeping") end)
end

In this case we are starting a simple process in our callback that just sleeps for one second and then outputs something - this is enough to satisfy the API of the start callback but we don’t see “done sleeping”. The reason for this is that by default mix run will exit once that callback has finished executing. In order for that not to happen you need to use mix run --no-halt - in this case the VM will not be stopped.

From what I am gathering, I will be responsible for writing something which halts exit? It is the fact it is exiting by default that is throwing me off.

If I am planning to build long-running apps, what is the suggestion? Can I use mix run --no-halt in development and distillery to create a release for production?

6

In a production deploy the application is started in it’s own process. mix run simply runs the application startup just like any other code but then exits because it is done running the code. The process of launching the application in production is entirely different. You do not use mix in production.

7

I won’t be using mix in production. I’ll be using distilleryto create a release and running through bin/myapp foreground etc.

But in the case of development, how do you run long-running processes for your app? I’m currently using mix run --no-halt which is working fine for my purposes, but am now interested how others are handling this.

8

As I stated before - typically you would be using iex -S mix so you could observe and interact with the running application directly.

Supervisor and Application: Starting Applications (You may want to start at Introduction to Mix)

1 Like
9

Apologies! Got carried away on the SO post.

Thanks for the clarifications. I’ll go through the getting started guides before posting more questions.

I don’t think it’s helped that I’ve taken on Phoenix at the same time.

10

Using --no-halt is standard (unless you want the shell with iex -S mix). Even Phoenix’s phx.server adds --no-halt internally: https://github.com/phoenixframework/phoenix/blob/v1.3.0-rc.2/lib/mix/tasks/phx.server.ex#L27

So don’t worry about it, that’s the correct usage. In production you would use e.g. distillery as you have said.

5 Likes
11

I’m not sure that is the core issue - it probably has more to do with the “I want to do Rails/Sidekiq in Elixir/Phoenix” approach. Lots of people approach Elixir/Phoenix with a “I learn best by doing a interesting project” attitude, not allowing for the fact that their current mindset may not be ideal for working with Elixir/Phoenix. This is a theme in Cameron Price’s Micropatterns: Learning to Reach Quickly for the Right Tool - where he found out that for him personally it was necessary to do lots of little exercises before his mindset shifted sufficiently before it made sense to attempt a larger project.

Now I know nothing about either Sidekiq or Verk but from scanning Verk’s documentation I’m getting the impression that the intention is to insert Verk into your application’s supervision tree - much like Ecto is running in the supervision tree of the application that is running Phoenix - so you are also confronted headfirst with OTP.

This is where another common point of confusion comes into play:

mix new hello_world

creates a library application, while

mix new hello_world --sup

creates what you more conventionally would think of as an “application”. A library application doesn’t implement the application callbacks and as a result can’t be started or stopped. Most library applications will however still autonomously manage their own processes once they are included in the primary application’s supervision tree.

So it seems to me that Verk’s primary intent is to be run in the same primary application that is already running Ecto and the various bits that Phoenix relies on like Cowboy.

I’m used to having app and worker processes separated and scaling them separately. As such, I might have one app server on one box and 3 workers on other boxes.

Now this describes a somewhat different situation. Usually a “box” is running a node and it’s the node that is running your primary application. So each “box” runs it’s own node with applications running in it. Your description sounds like you intend to run Verk on a node that is separate from Phoenix (possibly even Ecto as part of yet another application on yet another node). This can be accomplished with distributed Elixir but interestingly:

Sorry if I came across as discouraging the use of --no-halt. To me it just seemed more advantageous to seize the opportunity to shed a bit more light of what an “application” actually is in the Elixir world rather than sweeping it under the carpet with --no-halt.

2 Likes
12

Thanks for the write-up.

This is a great approach, but I largely learn by taking on more than I can chew, by drowning myself in complexity and overwhelming myself with lots of new concepts and languages and frameworks. This gets me to the most problematic questions very quickly, which I then search on, and, if not fruitful, bring to the community (hence being here). I can suss out most of the small things on my own. This brute-force approach has served me well for the last 8 years, but is of course not without its downsides, and, I will say, I have definitely jumped the gun by asking questions here without having even gone through the getting started guide. No excuse for that.

But… though I am taking on Elixir, Phoenix, OTP, Docker, Kubernetes and quite a substantial project, I still have enough experience to lean back on which has significantly reduced the learning curve – and, sadly, as is understandable, had me drag that past experience into new areas, perhaps where it doesn’t belong. No doubt I’ll have to rethink how I program, even if not just for the reason that I’m taking on my first functional programming language.

Either way, Verk, or some kind of worker process is critical to the underlying feature-set of my application(s), and usually one of the first problems I have to solve when embarking on any new non-trivial web application. I’m very happy with the current state of my extremely modest but expanding codebase. It is vastly simpler than counterparts I’ve built in other languages/frameworks, and it is implicit in all the right ways.

I’m now diving into the supervision tree, OTP and the rest to better understand the underlying architecture – this is crucial, anyway, for me to configure BEAM to work across Kube nodes with auto-discovery… but the fact is I don’t get excited until I can see myself building something wide-ranging and useful. I was only temporarily happy with using --no-halt and being done with it just because it removes my current roadblock. I’d definitely be returning to the issue to ensure I’m not doing anything stupid.

Verk is framework agnostic, and there isn’t a mention of Phoenix in the docs or code or even in the issues (apart from a ticket I created). I’m currently switching the supervisors that start with the application based on an env var (“APP_WORKER”). If present, it will load Verk.Supervisor, and any other dependent supervisors (Ecto.Repo), and if not present it loads my default Phoenix supervisors, along with Redix to enqueue jobs from my app.

I’m not entirely happy with that. There was a suggestion in another thread to separate out the domain/data layer and share that code between two completely separate applications – and I have heard I can add a whole other application as a dependency. This is stuff I’ll be researching a little more. My current setup isn’t ideal, but it is a pattern that works for many production systems, and, though a little tangled, is simple to maintain. However, I would like to separate the worker dependencies, because my worker doesn’t need Phoenix and a few other dependencies that my Phoenix app relies on. If it becomes a serious problem when I start to deploy (within the next week), then I’ll rethink the whole structure of it.

Exactly… I should be careful about my wording! That’s right, I’ll have Verk in a pod that will be deployed to any number of nodes, and I’ll have my Phoenix app in another pod. They will only speak to each other through Redis, my app will push data to redis and Verk will pull, which will then use my models in my app to run business logic and query the database.

But I can see how a distributed pure Elixir setup might accomplish this. Likely, in the future, I’ll attempt to deploy a pure distributed Elixir version of my architecture and see how it fairs. But I just don’t think it will offer the kind of tooling I have with Kubernetes, and Kube is something I can use to manage clusters of any kind of application, not just Erlang-based. And there is no substitute in pure Elixir for a durable worker system. To me, that’s reason enough to stick with my current setup.

Absolutely, and I really appreciate the breakdown. There are quite a few new concepts I need to get my head around, but I’m having a lot of fun - currently with simple pattern matching and piping. Just trying to shed old OOP habits, and am already starting to see the advantages.

One of the reasons I said learning both Elixir and Phoenix at the same time might have been a classic mistake (same mistake I made with Ruby/Rails), is because I didn’t realise application was an elixir concept. Again, better run through those getting started guides!

13

More accurately application is an OTP concept that is exposed through Elixir’s Application module. You might look at:

the first few paragraphs of learnyousomeerlang.com/building-otp-applications#why-would-i-want-that

erlang.org/doc/design_principles/applications.html

https://hexdocs.pm/elixir/Application.html

2 Likes
14

Cheers. I’ll run through those links!

15

To shed a little bit more light on it:

The terms OTP ‘Application’ and OTP ‘Process’ had me confused for quite some time when I started out working with Elixir, because they feel very different from operating-system-level applications and processes.

But here is the thing: When running Erlang or Elixir code, the Open Telecom Platform/the BEAM Virtual Machine is the environment the stuff runs in. That this might in turn run on top of another operating system (Which is not always the case – see Nerves or Xen for instance) is not important.

The service your ‘OTP System’ (Which might run as an Operating-System level application) provides might be comprised of one or more OTP applications, which can be started/stopped independently.
OTP processes also work very similar to operating-system-level processes, in the sense that they are scheduled and can pass messages (Although Erlang’s message passing API is a lot cleaner than managing Unix pipes :stuck_out_tongue: ), but of course they are far more lightweight than OS-processes.

1 Like
16

After reading through some of these links… it seems like I might be able to have a completely separate application for my worker, and reference my Phoenix application as a dependency? – but I’d want to do that without running the application; using it as a library application.

17

Yeah @Qqwy is dead on here. The BEAM VM is more like the hardware of a system. OTP is like the OS. Processes are like Processes on a real OS. Applications are like real applications on an OS (groups of processes). Etc… etc… etc… :slight_smile:

3 Likes