Learning, unlearning conventions

[taro]

Learning Phoenix and its v1.7.0-rc for the first time, I’ve made arbitrary changes to phx.new conventions, mostly merging, colocating, and simplifying. As a learner, I needed it simple.

Because I had no supervision, I wanted to report what and why I did here. And hopefully I’ll see room for improvement from your feedback.

• Colocating verticals: related entities, liveviews and tests under single directory
• Domains: *_web/*.ex are merged to related Contexts
• Entities: schema, changeset, queries, repo calls in a single module
• top-level assets/ and rel/ goes under priv/
• "app" denotes general, main context
• Medium-like auth based on access to email address

Colocating verticals

@atomkirk

The main thing we learned was to organize your project around features, not functionality.

App, web, and test are divided too early in the file system when many of them corresponds 1 on 1. Closest ideas were separated the farthest. They could be separated neatly into different modules, but close in the project structure.

- lib/demo_web/
- test/
! lib/demo
! lib/demo/domain/*.ex
+ lib/demo/domain/*.live.ex
+ lib/demo/domain/*.test.ex

Domains

Yes I made the name up. Domains have a similar concept to Contexts. .ex files under lib/demo/ and a directory of the same name. But now they don’t call Repo as much, and accommodate web stuff. They export plugs and on_mount callbacks.

  lib/demo/domain/
+ lib/demo/domain.ex
+ lib/demo/other_domain/
+ lib/demo/other_domain.ex

The rules around domain are

• Only domain.ex and siblings call domain/*.
  • domain/* don’t call other_domain/*
• domain.ex can expose domain/*.
• domain/* can use all ../* (siblings of domain.ex).

Entities

@mathieuprog

From the “Programming Ecto” book:

As a general rule, we recommend putting the pure functions that manipulate
queries, changesets, and multis into their associated schema modules.

One thing I didn’t like about Context is that it gets bloated so fast with all the trivial queries and repo calls from its many schemas. So I have them distributed to their schemas and call the module just “entity”. If queries get complex, they will need a dedicated module.

- Demo.Context.get_this/1
+ Demo.Domain.This.get/1
- Demo.Context.get_that/1
+ Demo.Domain.That.get/1

# domain.ex
alias __MODULE__.{This, That}

The rest

I didn’t wanted shallow, insignificant directories oozing out to the project root. So I put them in priv/. I think it makes sense categorically. With test/ dissolved in lib/, it was just assets/ and rel/.

- assets/
+ priv/assets/  
- rel/
+ priv/rel/  

With the structure, I’ve put "app" when I need a general naming. application.ex became app.ex with app/. And config/config.exs became config/app.exs to avoid meaningless repetition in path.

- config/config.exs
+ config/app.exs
- lib/demo/application.ex
+ lib/demo/app/
+ lib/demo/app.ex

This is something else, but I changed phx.gen.auth into an email-only authenticatIon process. It’s a much lighter auth solution that will cover many use cases.

The top module

lib/*.ex inherits lib/*_web.ex and continue to define interfaces for other modules. I’ve renamed them and removed unused interfaces according to the new scheme.

Demo
  .domain/0 # controller/0
  .entity/0
  .live/0 # live_view/0
  .user_interface/0 # html_helper/0, I find `html` too narrow.
  .routes/0 # verified_routes/0, the word already means established paths between two points

static_paths/0 is moved to config/app.exs and refered in Demo.App.static_paths/0.

Project structure example

Anything in straight line is sibling, diagonal is child.

demo/
  config/
    app.exs  dev.exs  prod.exs  runtime.exs  test.exs
  lib/
    demo/  demo.ex
      app/  app.ex
        components.ex  home.live.ex  layouts.ex
        error.ex  error.test.exs
      auth/  auth.ex  auth.test.exs
        access.ex  access.test.exs      
        account.ex  account.live.ex  account.test.exs
      endpoint.ex
      gettext.ex
      mailer.ex
      repo.ex
      router.ex
      telemetry.ex
      test_helper.exs
  priv/
    assets/  gettext/  rel/  repo/  static/

Now, teach me.

[trarbr]

The priv directory is kind of special, as it is included in the release. For that reason, you usually only put stuff there which has to be there for the app to function. So I would not recommend putting assets and rel folders in there.

About the rest, I have no strong opinions. If you stray from the defaults it may be a little more work to onboard people who are used to the Phoenix conventions. But if it works well for you, then there should be nothing stopping you. Phoenix has conventions, but it is also very flexible.

[cmo]

I don’t see the point in renaming config.exs. base.exs would be a more appropriate name imo but you’re just creating confusion by changing it. Same goes for app.ex. Convention is convenient.

assets gets built into priv andrel is for releases. Have you installed some js libs, built and deployed this?

[BartOtten]

@taro

App, web, and test are divided too early in the file system when many of them corresponds 1 on 1. Closest ideas were separated the farthest. They could be separated neatly into different modules, but close in the project structure

I see your point but many editor (plugins) can enable switching between modules and their tests by using file name convention. Your code structure would need a specific (extra) config for the plugins which ‘just work’ for other Phoenix projects.

@taro

• Only domain.ex and siblings call domain/*.
• domain/* don’t call other_domain/*
• domain.ex can expose domain/*.
• domain/* can use all ../* (siblings of domain.ex).

Also, have you looked at the Boundary lib? I know from experience that the use of path structures to ‘limit what can call what’ usually becomes problematic in larger projects. Boundary helps a lot to still keep things customisable but sane.

[BartOtten]

The author of Boundary, Saša Jurić, also has a brilliant blog history about maintainable Elixir; it covers some of your changes. Towards Maintainable Elixir: The Core and the Interface | by Saša Jurić | Very Big Things | Medium

[sodapopcan]

I agree with splitting by features over functionality, but there is a line with how far to take this. Phoenix draws a line (there are other lines to be drawn from there) between your business domain and the web domain which for me in my experience is “correct” (ymmv). Within the web domain, LiveView does the same in bundling up view/presentation and functionality by feature.

The business and web separation is useful in a several different ways. The main thing for me is that I want to treat my business domain as a kind of toolkit for building UIs around, possibly completely different clients (web, api, mobile, etc). Phoenix provides a little clue here in that the web namespace is MyAppWeb as opposed to MyApp.Web, but I think this elludes many people. `There is a thread here somewhere where someone puts it well by saying: “Think of the REPL as another one of your clients”. I find this to be a really nice balance of being flexible without getting into solving problems I don’t yet have. Now even if you know that you’re only ever going to have a web interface, it still isn’t always going to map 1-to-1 with your business logic which you even alluded to. There might be another part of the site where you want to reuse some some domain logic so it wouldn’t make sense to mix them. Finally (well, there are probably more reasons but I’ll stop here), having the domain logic categorizes all the “backend” functionality together making it very easy to browse and find stuff.

As for tests, I personally would never co-locate them. I’m a huge proponent of testing but I don’t take the “the tests are part of our app” stance; I think of them as yet another client. They also don’t always map 1-to-1, especially end-to-ends. Dave Thomas did give a good talk on the directory structure being wonky. It’s something I had thought about myself before but I’ve never found it to be a hindrance. It’s one of those things that if it changed I’d be like, “Oh ya cool, this is a little better” but I really don’t think it’s as big an issue as some make it out to be. And I certainly wouldn’t call any of the top level directories insignificant.

I have plenty more to say on this topic (especially around messy context files… they don’t need to be!) but that’s probably enough for now and my dog needs to go out. I will say, though, that Phoenix is very flexible and you have to do what makes sense to you! There are advantages to at least somewhat adhering to what’s suggested for us, especially if you’re working for a company and want to onboard people quickly, but it’s by no means necessary. I’m more just trying to shed some light on why things are the way they are and why I actually quite like it the way it is.

[sodapopcan]

One other thing I thought of on my way-too-short dog walk that I actually often think about creating a lib/my_app/domain directory to separate the business logic from the application logic. One glaring thing that I feel is off about the default structure is that as a rule is that stuff in lib/my_app should never reference anything inside lib/my_app_web yet the MyAppWeb.Endpoint is passed to MyApp.Application’s supervision tree. I believe this was done for convenience and simplicity with how Elixir Applications work, but it still raises a brow.

[al2o3cr]

@taro

Now, teach me.

Why? Since you understood the reason for all the conventions you decided to override, you’re plenty familiar with the tools.

You did understand those, right?

[taro]

I don’t know much reason behind the conventions. I’m teaching myself with no technical background or environment other than online community resources. I’ve barely learnt the basics of Elixir and Phoenix in the last month. Now I’m trying to make sense of the tools and get familiar, but far from plenty.

Elixir and Phoenix conventions are accumulation of the last decade. I try to follow up, searching for discussions and reading books. But still, some of it feels arbitrary to me. So I align things to make a better pattern. Then I worry messing them up. That’s why you could teach me. I’m reading precious feedback over and over again to reflect and make the next decision.

[sodapopcan]

Elixir and Phoenix conventions are accumulation of the last decade.

It goes a little farther back than that. From a newbie’s standpoint, I would say that the two biggest influences on Phoenix conventions that you should care about are Ruby on Rails and Domain Driven Design.

…except that you don’t need to care at Rails if you don’t already know it. I’m a former Rails developer and have a massive soft spot for it (I still think it’s a very viable option if you’re deep into OO), but Phoenix has done a lot of really good work since its inception to make itself less like Rails. The influence is still there and it’s not a bad thing because Rails is not all bad.

Domain Driven Design (DDD), on the other hand, is very pertinent to Phoenix. It’s a very large topic but Phoenix’s Contexts come directly from DDD’s Bounded Contexts. The DDD book itself is massive so I recommend reading the very digestible DDD Distilled as a gentle introduction. I’ve honestly never read the full DDD book cover-to-cover—that book literally has a bunch of paragraphs in bold and says, “If you feel confident and don’t want to read this whole book, just read the bold paragraphs”.

Of course, like any concept in programming, there is an army of people who are anti-DDD. Are they right? Who knows, but at least peruse the previously linked Martin Fowler short article for prior DDD art.

I’m also very sorry if I’m telling you stuff you already know (programsplaining? lol)

[taro]

Thank everyone for the feedback.

@trarbr

The priv directory is kind of special, as it is included in the release.

I didn’t know the whole priv/ is included in the release. I’m taking them back. I tried to hide them mainly because assets/ is such a generic, confusing name for web artifacts. Is it a common practice to call JS and CSS “assets”?

@cmo

base.exs would be a more appropriate name

I’ll adopt config/base.exs. config/config.exs confused me initially. It’s the whole directory that makes configuration. “base” sounds exactly like what the single file does.

assets/ in priv/ builds just fine with right configuration. But I’m taking it back to the root. I just hope for a better name for it.

@BartOtten

Boundary helps a lot to still keep things customisable but sane.

I have a feeling that this library will benefit so much! Especially with non-conventional project structures. Thank you so much for this one. And the author Saša Jurić . I’ll take a good look and adopt asap.

As for conventions and editors, it’s a win for me if it’s achievable with extra config.

@sodapopcan

The business and web separation is useful in a several different ways.

It all makes sense. Thanks for the elaboration. My naive but sincere question would be

Why can’t they be separated module by module sitting side by side?

instead of lib/* level. BEAM is file structure agnostic anyways. Things go by modules. Can’t we reap all the benefits of separation and colocation? High cohesion, loose coupling. If I have three interfaces and add a feature. I’ll need to work across them anyways.

group/
  new_feature.ex
  new_feature.live.ex
  new_feature.cli.ex
  new_feature.mobile.ex

I’m not saying every single thing maps 1-to-1.

loose_grouping/
  i_map_one_to_one.ex
  i_map_one_to_one.live.ex
  i_map_one_to_one.test.ex
  i_don_t_but_i_belong_here.ex
app_or_general_group/
  i_go_over_many_groups.ex
  i_am_shared.ex

@sodapopcan

The DDD book itself is massive so I recommend reading the very digestible DDD Distilled as a gentle introduction.

This is nice, I’ll definitely read it.

One thing about “Domain” confuses me. Sometimes it’s plural “business domains” as in business verticals such as engineering, marketing, HR, finance. Other times it’s singular and horizontal like “business domain” vs “web domain”.

[LostKobrakai]

@taro

I’ll adopt config/base.exs. config/config.exs confused me initially.

That won’t work. config/config.exs and config/runtime.exs (and the deprecated config/releases.exs) are the only files mix knows about and those names are hardcoded. All other files in the config folder are convention, optional and need to be included explicitly by code from config/config.exs. So it really only is a base config if you make it that.

[taro]

I’m confused. It seems it’s working with

defmodule Feder.Mix do
  use Mix.Project

  def project do
    [
      ...
      config_path: "config/base.exs",
      ...
    ]
  end

  ...
end

How can I tell apart if it’s not working? I see no sign.

[LostKobrakai]

Oh, didn’t know there were actually configurable (especially given there’s not just one file).

[taro]

UPDATE

What I was trying to do turns out to be called “Feature Slicing”, or “Vertical Slice Architecture”. From my short research, it seems

• Great for Agile/Scrum, some say it’s the only way.
• Suitable for small teams with distinctive roles, as opposed to well-orchestrated large group of developers.