Thanks for the write-up.
This is a great approach, but I largely learn by taking on more than I can chew, by drowning myself in complexity and overwhelming myself with lots of new concepts and languages and frameworks. This gets me to the most problematic questions very quickly, which I then search on, and, if not fruitful, bring to the community (hence being here). I can suss out most of the small things on my own. This brute-force approach has served me well for the last 8 years, but is of course not without its downsides, and, I will say, I have definitely jumped the gun by asking questions here without having even gone through the getting started guide. No excuse for that.
But.. though I am taking on Elixir, Phoenix, OTP, Docker, Kubernetes and quite a substantial project, I still have enough experience to lean back on which has significantly reduced the learning curve – and, sadly, as is understandable, had me drag that past experience into new areas, perhaps where it doesn’t belong. No doubt I’ll have to rethink how I program, even if not just for the reason that I’m taking on my first functional programming language.
Either way, Verk, or some kind of worker process is critical to the underlying feature-set of my application(s), and usually one of the first problems I have to solve when embarking on any new non-trivial web application. I’m very happy with the current state of my extremely modest but expanding codebase. It is vastly simpler than counterparts I’ve built in other languages/frameworks, and it is implicit in all the right ways.
I’m now diving into the supervision tree, OTP and the rest to better understand the underlying architecture – this is crucial, anyway, for me to configure BEAM to work across Kube nodes with auto-discovery.. but the fact is I don’t get excited until I can see myself building something wide-ranging and useful. I was only temporarily happy with using --no-halt and being done with it just because it removes my current roadblock. I’d definitely be returning to the issue to ensure I’m not doing anything stupid.
Verk is framework agnostic, and there isn’t a mention of Phoenix in the docs or code or even in the issues (apart from a ticket I created). I’m currently switching the supervisors that start with the application based on an env var (“APP_WORKER”). If present, it will load Verk.Supervisor, and any other dependent supervisors (Ecto.Repo), and if not present it loads my default Phoenix supervisors, along with Redix to enqueue jobs from my app.
I’m not entirely happy with that. There was a suggestion in another thread to separate out the domain/data layer and share that code between two completely separate applications – and I have heard I can add a whole other application as a dependency. This is stuff I’ll be researching a little more. My current setup isn’t ideal, but it is a pattern that works for many production systems, and, though a little tangled, is simple to maintain. However, I would like to separate the worker dependencies, because my worker doesn’t need Phoenix and a few other dependencies that my Phoenix app relies on. If it becomes a serious problem when I start to deploy (within the next week), then I’ll rethink the whole structure of it.
Exactly.. I should be careful about my wording! That’s right, I’ll have Verk in a pod that will be deployed to any number of nodes, and I’ll have my Phoenix app in another pod. They will only speak to each other through Redis, my app will push data to redis and Verk will pull, which will then use my models in my app to run business logic and query the database.
But I can see how a distributed pure Elixir setup might accomplish this. Likely, in the future, I’ll attempt to deploy a pure distributed Elixir version of my architecture and see how it fairs. But I just don’t think it will offer the kind of tooling I have with Kubernetes, and Kube is something I can use to manage clusters of any kind of application, not just Erlang-based. And there is no substitute in pure Elixir for a durable worker system. To me, that’s reason enough to stick with my current setup.
Absolutely, and I really appreciate the breakdown. There are quite a few new concepts I need to get my head around, but I’m having a lot of fun - currently with simple pattern matching and piping. Just trying to shed old OOP habits, and am already starting to see the advantages.
One of the reasons I said learning both Elixir and Phoenix at the same time might have been a classic mistake (same mistake I made with Ruby/Rails), is because I didn’t realise application was an elixir concept. Again, better run through those getting started guides!
), but of course they are far more lightweight than OS-processes.
