Discussion: Don't add a database layer to your Phoenix application

Yep, you have to package things up as a Gem, whereas in Elixir, you can just include any standard mix new app via that one line. That’s why it blew my mind, because it was so easy :lol:

By that definition, Python is the Nr. 1 architecture pusher… Have you ever looked at cookiecutter? It’s probably one of the best tools to generate projects for any language (just look at the list of templates). Not as powerful by default as a Mix task (by design), but makes it much easier to create project templates. It can be used as a library to create “tasks” that are as powerful as Mix tasks, too.

I don’t think you should call this “pushing architecture”. It’s just a template, and a very weakly enforced one, too… It’s a text file: you can customize it beyond recognition, even changing such things as supervision trees. If you want to see an example of someone pushing (bad) architecture, take a look at any WSGI python framework (i.e. Django, Pyramid, Flask, etc.). They are by design constrained in way that the application must be designed in a certain way. I only call this architecture bad because it makes it very, very hard to use stateful connections (which would be unacceptable on the BEAM world)

Yes, and then everything is so decoupled that you end up surprised when the Web part of your application starts without your database, or something like that. Your Project turns into a decentralized Frankenstein monster, in which parts of it move by themselves, disconnected from the central authority of the brain. People here seem to have a passion of decoupling everything into its own app, which is not bad in itself, but maybe some things are meant to be coupled. Otherwise, you might get questions like these (from the subreddit): https://www.reddit.com/r/elixir/comments/6zvsxd/avoiding_circular_dependencies_in_a_phoenix/

I’m nowhere near experienced enough, and maybe I shouldn’t even participate in a discussion that includes all these experienced people, but sometimes this whole “Decoupling” business starts looking less and less like an engineering practice, and more like a religious stance.

I haven’t used that many languages in practice, but for my project I read the “propaganda” for almost everything that’s there (Smalltalk, PHP, Java, Scala, Clojure(script), Racket, Common Lisp, other Lisp dialects, Haskell, OCaml, Python, Rust, C++, Erlang, LFE), and even tried some of them, until settling on Elixir. They all think they have everything figured out.

In my opinion, saying things such as “Only Elixir knows the right way” are counterproductive and sound like propaganda.

I also think that Elixir might be close to a monoculture now, with “everybody” using the same things (Phoenix, the default generators, etc), but if it grows ot might become more like Python: multiple ways to do something, each with its own pros and cons.

4 Likes

I think there was something like that for Rails once :lol:

Maybe architecture is the wrong word. It’s just best practices/methodologies.

That’s the sort of thing I wanted to learn about when posting this thread :slight_smile:

I’m in the same boat as you, but to me, when I think of the kind of applications I want to build having separate, independent components seems to make a lot of sense to me. For example, in a social networking app with potentially millions or hundreds of millions of users you’d want the User registration side to be an independent system so that it can be individually scaled/secured/backed up frequently, while other components of the system may not be quite as ‘important’.

I don’t think anyone is saying only Elixir knows the right way. It’s more about the choices it makes and becomes a proponent of :slight_smile:

I like that unity :blush:

Right with that said, I’m off back to Dave’s course - I’m dying to finish it :lol:

Architecture is not how files are laid out in the filesystem. It’s how they interact at runtime. Very different.

3 Likes

By architecture I meant how the overall system is put together… coding principles (or patterns) would have been a better phrase to use :blush:

I just watched it again :slight_smile: It is a great talk.

There is more information on why you named it Context and the only thing really found why you named it at all was (paraphrasing): “We wanted a way to communicate the importance of API boundaries to our users”

Just to be clear I think API boundaries are a honking good idea and I very much like the way phoenix have developed over time. I just have a feeling by naming it you gave people a “hook” to hang their complaints on :smiley:

I still disagree. Coding principles and patterns should be more concerned with code in motion than code at rest. Patterns (at least in the Gang of Four sense) are all about how different objects interact at runtime and never specifies what directory structure they should be stored in.

It’s easy to get bogged down in choosing what file to put some bit of code in, and that is important, but it’s secondary to understanding how the code interacts while it is running. That’s where runtime dependencies/coupling (and runtime errors) come from. Not from the filename or directory they are stored in.

3 Likes

By “how the overall system is put together” - I mean how the app is coded (and I guess, how it is designed and will run, in the sense of processes etc). Was it developed using good coding principles?

I actually borrowed ‘coding principles’ from one of Dave’s posts in this thread (included below). To me you apply (hopefully good) coding principles when developing your app/project/system. Whether that is how your app is put together, naming conventions or the patterns or methodologies you use within the actual code (or how it operates when in use - which is what I think you are referring to).

There is more to a system then code, such as resources (likes databases, etc). If I have an application that uses my database and maintains a connection pool of connections I may only want to run this app on 2 nodes, yet have code on a multiple of nodes rpc into the 2 nodes holding on to the resources.

1 Like

I think what I’ve learned through the year from erlang is that you want to separate “services” in OTP application. With “service” here I mean things like queues, databases-access, caching layers and similar and don’t use OTP apps as a way to modularize code.

A separate OTP application also runs its own supervision tree and this is important to remember in terms on how you want to separate failures throughout your application. I’ve seen a tendency in elixir that people prefer to include services in their own supervision tree and in my opinion its sort of a slippery slope to coupling. I also think it makes it harder to trouble-shoot in production.

I think this ties in a bit with what you say about “code layout is different from how services interact”.

One “rule” I found useful to help with refactoring is to hide all functionality of an “entity” behind a clearly defined API which must not leak implementation detail. Once you have a small, well-defined entry point into your application it is easy to move between various ways of doing code layout.

5 Likes

Sounds like we’re in agreement on this point, as long as “coding principles” is more about code in “motion” at runtime than at “rest” on the filesystem. :thumbsup:

There’s definitely advantages to organizing your code well on the filesystem, don’t get me wrong. But those are secondary to how the code interacts in memory at runtime.

2 Likes

Minor caveat: library OTP apps don’t have a supervision tree, but are just modules ready to be called when necessary.

I’m of two minds on managing supervision trees. Yes, I like the idea of something pre-baked out of a dependent app and ready to go all at once just by adding it to deps. OTOH, configuring it can be a nightmare. I wonder if there’s a middle ground where the dependent app provides the hook to a whole (complex) supervision tree, but the host app manages config and starts up that whole sub-tree with explicit config under its supervision tree. I need more hands-on experience to have a strong opinion in this area, but that is the way I lean right now.

2 Likes

I think there are so many interesting ways to manage persistence, some enabled by the BEAM and some less so, that it makes a lot of sense to extract the database layer from the web layer. A concept I am currently looking into is virtual actors.

I’ve been following this thread with interest and this comment summed up a lot of what I think. Simply by starting with the a SQL database as a default phoenix is quite railsy on the whole spectrum of possible Elixir applications. However I don’t want to dwell on that point. The whole spectrum of possible Elixir applications is enormous.

What I do believe is that a healthy ecosystem has multiple projects that are tackling similar problems. It is why I continue to work on my alternatives to Cowboy, plug and phoenix (I have linked to them before so don’t need to here). For context at work I have an application that is using cowboy, plug and phoenix and it is working very well.

Where Phoenix can choose productivity by reusing concepts and conventions familiar to many developers an alternative can through away familiarity and explore the alternative further. A single project can never really optimise for approachability and innovation.

To answer the original question to this thread. I think with phoneix as it is now; using the --no-ecto flag and immediately adding ecto to a sister application within an umbrella is a nice idea but probably not worth it on day 1 of a new project.

1 Like

This is precisely what the two new generators do:

mix phx.new.web

and

mix phx.new.ecto

Just create umbrella app, cd apps/ and run these and you’re set.

1 Like

Or even in just one step with

mix phx.new --umbrella
3 Likes

Is phoenix still describing itself as just a “web layer”? Here is a generator that looks like it generates nothing for the web, I assume?
Thanks anyway, I did not know about the new generators and will look into them a little further before commenting any more

I don’t know how it describes itself, but this is generator from Phoenix, that does generate a pure Elixir app with Ecto dependency. There is no dependency on anything Phoenixy in it.

In his course PragDave describes it as “more than a web framework” - in fact he starts the section by saying what Phoenix is not. “I don’t believe Phoenix is a web framework - it’s way more than that” :003:

This topic is amazing.

A few things i’ve noticed:

People are taking architectural patterns (microservices/onedbper/etc) that were devised in the context of another system language and interpreting them as if they could be mapped into erlang/elixir.

I contend that BEAM is better thought of as an Operating System environment than being mapped to the UNIX context of system design ( ie: using JVM or GO to ammend the operating system and provide additional layers of abstraction)

In BEAM we are given the tools of an Operating System - Processes, Message, Monitors , etc and in that context we are also given the capability to abstract, layer and isolate our processes and services inside a distributed operating system.

I establish that it’s essential to understand the difference between the container and the content when looking at patterns that have been established for other languages and systems - and the same step is necessary when developing in Elixir Erlang.

I’m saying this because in all my other language and system development experience - having an operating system that is actually working to keep your code running as fast as possible - with as much resources as required - while maintaining a variety of system invariants is alternative to the design of the timeshare unix OS.

Amazing! Here we have an distributed operating system who’s natural building blocks are modules, OTP applications and processes.

Part of the richness of Erlang/Elixir is exactly this and so it behooves us to help guide new and burgeoning developers into the richness and differences of this platform.

The work that PragDave is suggesting to his students is an essential part of this.

Instead of citing patterns that were designed with other constraints in mind - start to discover the new (old) frontier that BEAM is offering!

This could mean discovering effective ways to group and organize code around a cluster, composing and isolate various services inside the context of the language.

Instead of having to fall back and forth between the context of the external timeshare OS and the semantics of a shared memory timeshare process (like a go microservice) that is being externally co-ordinated around other timeshare OS processes ( like a database process ) using a scheduler that is based on the general approach that the machine must be protected from the processes - the OS knows nothing of the critical nature between it’s two processes ( aside what a devop might instruct via runit or it’s cousins)

The BEAM approach is that the ‘OS’ should protect it’s processes, deal with faults and return itself to a known state - where the known state is it’s particualr software and not - in the case of timeshare-os - default form designed to generally run various processes.

This is what makes me excited about erlang/elixir.

Glad for the conversation!

Best,
Curtis RobotArmy

4 Likes

It would be awesome if this tool gained some traction.

2 Likes