Best practice for directory structure of a larger application?

Questions / Help
, ,
1

Hey,

I am very new to the Elixir world, but doing software development for many years. The Elixir ecosystem looks very promising to me. Currently I am not sure how to structure a larger application in an optimal way. Let’s say I want to create a web shop application named “shop”, which is an API only (REST or GraphQL) application. I would structure it this way (disregarded Elixir):

  • shop
    • customers
    • payment
      • paypal
      • stripe
        :
    • communication
      :
      .

With Elixir (Phoenix) it would be?

  • shop
    • lib
      • shop
        • customers
        • payment
          • paypal
          • stripe
        • communication
          :
      • shop_web (all the web/api stuff)
        :

But what about an Elixir structure like this:

  • shop
    • lib
      • customers
      • payment
      • communication
      • api (all the web/api stuff)
        :

That would be more “natural” to me (for an API only application). Otherwise I would have repeated “shop” and all the code is split more or less in two main directories: “shop” and “shop_web”

Another question: Is it uncommon to have the application name in the module identifier/path? So it should not be “Shop.Payment.Paypal”, but just “Payment.Paypal”. Right?

How should the Ecto repo modules be named? “Payment.Paypal.Repo”? Or “Payment.Paypal.DB”? Should the Repo be seperated from related business logic?

Sorry for mixing the topics a little bit, but for me it’s somehow connected :wink:

2 Likes
2

@cvh23 You really need to learn lots of Phoenix stuff. Many of your questions are from someone who never tried them - especially Repo naming which is included in every new phoenix project (mix phx.new generator). Please read official Phoenix documentation. After your first Phoenix app if your code would be really big you may consider refactoring your app into umbrella app.

Some useful resources:
https://hexdocs.pm/phoenix/
https://hexdocs.pm/phoenix/Mix.Tasks.Phx.New.html
https://hexdocs.pm/ecto/Mix.Tasks.Ecto.Gen.Repo.html

1 Like
3

Thanks @Eiji for the links. I’ve already tried a small project and some of the generators, but I am not sure how obligatory or recommended these settings are.

I’ve also read about umbrella apps, but I thought this approach maybe “too big” for an application which just consists of some parts/modules.

4

This is a template for phoenix projects. Nobody forces you to use it, but it’s a useful resource for generic use cases. It’s made by Phoenix core developers, so those are recommended. Hover using them because they were in some template isn’t enough. For example by default every phoenix project have PostgreSQL database. Depends on your project requirements some part of such code may change, but it’s mostly something like database configuration (like credentials) rather than whole application.

For new Elixir/Phoenix developers such template is more than enough as Phoenix is feature-rich framework (which in fact works like library). In order to develop applications well you should know concept like what are contexts and why we write them. All of this you can find in official documentation, ElixirConf videos and many, many more learning resources. Make sure you look also at Learning Resources category.

1 Like
5

One question first, is this for fun or profit?

Fun
Just experiment, take a look at the source code from changelog.com and hex.pm (search on github) and build what you think works for you and post it here for others to look at and help you with. It’s very easy to refactor in Elixir and you’ll learn a ton in the process.

Profit
Just build it with the default settings and ship it. No matter how you structure your app, if you don’t ship it it does not matter.

5 Likes
7

I’ve worked on a number of big Phoenix/Elixir applications. I led the development of a big one (200k+ lines) over 4 years. We did it both ways, the way phoenix boilerplate does it for about 2 years, and then we reorganized it. The main thing we learned was to organize your project around features, not functionality. Why? Because which is more common? working on all your tests at the same time? or all your controllers, views, or graphql code at the same time? Or, working on a particular feature? Sometimes you do refactor something across an entire category of functionality, but WAY more often you or someone maintaining your code is working within a feature/resource/group of resources. So here’s how we did it, and we never ever looked back. It it insanely amazing and other people who have worked on our project have commented on how much easier it is to work on a feature and find all the relevant modules they need.

lib/
  invoices/
    controller_tests/
      index_test.exs
      create_test.exs
      update_test.exs
      delete_test.exs
    controller.ex
    view.ex
    view_test.exs
    schema.ex
    schema_test.exs
    jobs/
     send_invoices.ex
     send_invoices_test.exs
  payments/
    controller_tests/…
    schema.ex
    …
  customers/
    …

We tore a page from golang and we put the tests right by the modules they test. This is so great. Why do we do all the work to build a mirror of our project dir in the test? Tests are code, they are part of our app. This also makes modules that are not tested stand out like a sore thumb. Need to update a module and the test? They are right by each other!

Anyway, I was stuck in analysis paralysis about this, asked the elixir community in slack if I should do this, they all advised against this, but this project has had this structure for 2 years now and I will be doing all projects like this in all languages/frameworks from now on. It is nonsense to organize your project by functionality (controller/view/etc) rather than feature/type/resource. I don’t know rails why ever sent us down that route.

Lastly, when you want to split it up into umbrella apps or microservices, this directly structure is exactly how you’ll want t do it, you’ll just copy the directory wholesale into another app/service/etc. This organization very very much helps facilitate a domain driven design mindset.

I highly highly recommend this :slight_smile:

21 Likes
8

At the moment fun, but with the perspective of profit :wink: Just looking how to realize projects with Elixir, which I have already done with TypeScript. For future projects I want to change my backend stack.

Thanks for the hints. Refactoring looks indeed a lot easier than in other ecosystems.

9

Thank you, looks interesting. In general I am also in favor of splitting apps into feature parts and not technical parts. Of course sometimes it makes sense the other way around.

10

In that case just start hacking :wink:

Not sure if you know but module names have nothing to do with the location of the file on disk. So even if your file is in shop/lib/shop/payments/paypal.ex you can still name the module MyApp.Paypal

1 Like
11

Interesting structure, not sure if I would like it but if it works for you, perfect!

How would you find files in your editor though, would that not be challenging with those filenames?

12

not at all! its awesome. In sublime or vscode I just do invoi/sche to get to invoice schema, etc. so its just the opposite of what you’d normally do mod/inv to fuzzy search for models/invoice.ex. Then, in vscode I do command + E and it shows that file in the sidebar and all the other relevant files I’m going to be working on are right there (tests, graphql resolvers, etc)

13

ultimately, because of how elixir modules don’t care about file location, elixir projects are extremely easy to re-organize. took me 2 hours to totally reorganize our project. So, just do whatever you like and if you realize later you want to do it another way, its super easy to change.

1 Like
14

This is really interesting … I may like this way … From your project structure I see that you are not using contexts, right? What’s with modules implementing shared functionality for multiple features? It could be a huge mess with a really big number of schemas, isn’t it?

However …

I don’t like putting tests in same directory. All of the modules in lib would be unexpectedly used in some cases like:

  • project documentation (@moduledoc false for all tests :icon_question:)
  • API of hex library (if not then it means you have 2 standards: one for apps and one for libraries which does not makes sense for me)
  • releases (if I remember correctly)

Also where is your test/support directory in such structure?

15

Yes, I know that. But I think its in general a good idea that name and path match :slight_smile:

16

What does your module naming look like?

17

Agreed, especially is large codebases

18

we renamed modules over time to match path, like Project.Payment (for project payment/schema.ex) or Project.Payments.Controller , but weren’t aggressive about it because it was really emotional more than practical. It FEELS good to have them match, but we rarely actually experienced a practical advantage. The only advantage I can think of is helping you find the file that contains the definition of a module when you are looking at the module being used. When I am looking for the file of a module, I just search for defmodule ModuleName and project search takes me right there. Or, more recently, most IDES with elixir language server have “go to definition”.

I agree though, if its not a big deal, its generally nice to have them match.

19

we did start moving to contexts with phoenix 1.4 come out and put them in payments/context.ex

so we had a shared directory that had child resources that had no clear home. so maybe shared/cloud_files/schema.ex (and others like context.ex). Cloud files being an abstraction of files stored on s3 or something. A lot of places in the app may use uploaded files.

So with tests, it works better than you may assume. None of those concerns you pointed out end up being a concern. I really think you should give it a shot and see.

we do still have a test directory that holds all the test support stuff (like test/support and fixtures, factories, etc)

If you only include your tests in lib and modify your mix.exs like this: test_paths: ["test", "lib"], because they are _test.exs files, they are ignored and not compiled into builds or releases.

2 Likes
20

Ah, I see … When I was asking I had in mind splitting project into reasonable parts (each in its own directory). For example lib/accounts/user/schema.ex, lib/accounts/role/schema.ex etc. In really big projects there are dozens (if not more) of ecto schemas, so it would be a small pain to scroll them to find the right one.

Following this article:

By default whole lib directory is send to hex. Do you have some pattern for files option to only include .ex files inside lib directory (asking as never saw it in practice)? I saw one pattern somewhere, but not sure if it works in this use case. I guess it was something like "lib/**/*.ex", right?

Oh, I expected that you would put fixtures and factories in lib directory as well. :smiling_imp:

Yes, I mean exactly this! As said I was not sure if they would be compiled or not. Now it makes more sense. Does ex_doc works in same way (i.e. only for .ex files)?

The more I read about it the more I’m interested!

It would be really helpful if you would provide all generic changes you made (like modification of mentioned by you test_paths). While I may sometime give it a try and slowly look for solutions/workarounds (for now I don’t have that much time) newbies may be confused seeing errors, so it’s important to properly describe needed changes. Maybe a blog post, small template or at least a name of such structure concept (like searching for Elixir Phoenix contexts in Google).

Also i have one more question … Did you had any problem with code analysis tools like credo. dialyzer or excoveralls? If so how did you solved them?

1 Like
21

yep, we did exactly this :slight_smile: I kept my example simple but we grouped mutliple resources into folder when it made sense. Your example was verbatum what we did :slight_smile: An accounts folder with users folders, auth folder etc.

I think you might have me here… I’m not sure how to make it work with hex. Your idea suggests there could be a way to make it work without too much trouble?

test_paths is literally the only change we made… other than that we just started putting _test.exs files next to the module they test and everything works exactly like it did before.

Nope, all of those tools work as expected without any special config/hacking.

I’ve worked on a lot of projects where tests are in a root test directory and without exception I’ve had headaches looking for the test that tests the code I’m looking at. It helps a lot when they maintain a proper directory structure that mirrors lib, but I hate maintaining that personally and no one does it perfectly. Often what I have to do is IO.inspect and then run the whole test suite to see what tests exercise the code in question.

1 Like