Discussion: Don't add a database layer to your Phoenix application

Well said, in general, but I’d like to focus in on this part a bit. What you describe there is the “web” part of the overall system (aside: we really should start thinking more in systems than applications in general).

I would argue that iex represents another user interface into the system. The user in that case is the maintenance programmer (including the original developer in a few months). I want to be able to test out some behavior easily. For example, suppose I want to create a new user for this purpose. It should be something simple like user = Accounts.create_user(%{username: "testuser", password: "secret"}). I should not have to figure out the exact params and simulate a call into the Endpoint just to create a new user from iex. This can be generalized to any other core behavior of the system.

Aim to make it easy to use your system from iex, and contexts become natural. Whether the context lives in one beam application or another becomes more an issue of code organization and keeping deployment options open. Getting the logical separation in place is the primary goal.

11 Likes

That’s very true - but I don’t think Dave, or anyone else is suggesting we can’t do what we want in Elixir ourselves (on the contrary, that Elixir actually makes it easier to do these extremely cool things!) but more that we have an opportunity, as a community (and as language and framework creators in the case of José, Chris and the core developers) that we can spearhead outstanding ‘modern’ development practices.

I find that extremely appealing as a user of these technologies and I genuinely feel that it is something that can help Elixir and Phoenix stand out even more from the crowd. It’s also already in line with how José and Chris think too - because they already spearheaded this movement with their work on Contexts (etc) :003:

Elixir and Phoenix could be the first language and framework where philosophy of and approaches towards software creation is as first class a citizen as is features.

1 Like

This is IMO one of the key points, not an aside :slight_smile: I like to use the term “web-facing system” (instead of a web app or web server) to denote a system which just happens to provide its service over the web.

I would argue that iex represents another user interface into the system.

This is an excellent point! Yet another interface are tests. Having contexts should simplify testing of the core logic (because we don’t have to deal with conns, requests, sessions, etc).

+100 :slight_smile:

10 Likes

I am pretty sure Smalltalk people thought the same about it. Then Java people. Then Ruby people. Then JavaScript people (sic!)

1 Like

Did they though - in the same way? Frameworks probably yes but languages themselves? (Notable exception possibly Elm and I’d guess Java… but I don’t think I know anyone who likes Java :lol:)

Ruby and JS (I don’t know enough about the others) don’t push you towards system architecture with things like single responsibility or microservices etc.

I think one of the advantages we have in Elixir is that José is part of the Phoenix and Ecto core teams and I think this lends itself very nicely to the kind of cohesion that I feel would be needed to pull this off :slight_smile:

Smalltalk. All the original GOF design patterns come from Smalltalk and it’s standard libraries. Ruby is basically Smalltalk so no difference here. JavaScript people think they invented reactive programming and what not. Erlang is a big one too. Lisp a great example too with powerful macros that allowed programmers to do crazy things with 5 lines of code (and a dozen of parentheses). Elixir while innovating is not the first to bring patterns and architecture on the table.

2 Likes

Do any of them push those architectures as part of their core? (I mean in the same way Elixir does with mix new)

What does mix new have to do with architecture?

In the same way that Phoenix pushes you down a specific path with its generators/Contexts.

So hmm. mix new does not do anything special except creating an established project structure in terms of files/directories etc. This is good to have standard like that but many other languages have something like that. Many others do not.

Compare that to Smalltalk, however, that does come with IDE/debugger/observer/image persistence/document browser/test runner thingy that does establish you way more infrastructure in place, and pushes way more conventions on you. Try Pharo Smalltalk, it is very educational experience.

I would disagree that it’s not doing anything special - it’s saying “Hey, look, this is a great way to structure your app - you don’t have to do it this way but we recommend it/think it would be easier/beneficial”. Hence my point that this could be capitalised on :slight_smile:

I don’t know about Smalltalk, or Java or any of those other languages but Ruby and JS definitely do not have this ‘guidance’ built in (or if they do have no where near the same kind of acceptance or adoption).

I would argue Elixir is already doing some of what I was suggesting (championing development in certain directions) - demonstrated by concepts such as this, Umbrellas and Contexts.

In that case, let me attempt to make that argument :slight_smile:

The vast majority of people, developers absolutely included, have a very low tolerance for diving into the deep end of a whole new world. People tend to do much better, and feel much better in the process, when they can incrementally move from A to Z rather than jumping over the other 24 letters in between. To the point that they will pass on methods that ask that of them.

No matter how much we would like this to be otherwise, it is a generalized truth about people.

The reason this matters comes from answering the question of why people come to Elixir and Phoenix. I don’t mean why they choose it over other options, but when they are already here … what are they trying to achieve?

I suggest that very very few are here to get a 5 year journey into a more ideal way of thinking about programming. I hope everyone does get that as a happy consequence of being here and using what are some phenomenal tools and getting to work with what is a great community of people. But that’s not what most show up for.

They want to achieve something very concrete, and often relatively trivial, today.

If we give them something that requires either too much learning (and let’s be realistic here: there’s already a lot to learn on the way in that door as it is!) or requires too much effort to put something useful together, they just won’t. We can point to Haskell and similar (great!) functional languages for the former, and Erlang (among others) for the latter as examples of these pitfalls.

We should also be realistic and realize that 90%+ of people using Phoenix will be making something small and simple. They don’t need complex or design perfection, they need a pragamatic tool to quickly get to their destination.

If they are successful with that, they may then move on to something more involved. This, I think, is a key point. How many times did the “we made a stupid prototype in half a day and basically got where we needed to be to make a full project out of it” meme occur in the Elixir Conference talks? I noticed it as a repeating story.

So it is key to have something simple, easy and “batteries included” that can get people walking (again, having to learn to walk in the land of FP, actors, and the rest in the first place). And that is why having something that provides all your pieces in one place is important to get started. Without it, people won’t start and few people will be there for the 5 year journey.

That said … most people don’t know they should be on that 5 year journey … perhaps until they are a year into it. (At least, that’s been my personal experience … :slight_smile:) But it is 100% worthwhile, and honestly the world of software development needs more of us to be on that journey.

So I do agree with you that there should be something better, something of a longer view. For bigger projects it is a requirement, for our personal development as writers of software it would be a blessing. But perhaps it could be delivered as a journey rather than an educational course.

What I mean is that we should perhaps be happy to have people walk in the door even if the anteroom isn’t where we want them to end up. Once they are there, there should be a nice and sensible path from that front door (i.e. Phoenix with Brunch and Ecto and all the other bits thrown together) to a better architecture, something like what you are advocating for.

And to me that is where the tools fall a bit short still: they give a great starting point, and you can go anywhere you want from there, but how amazing would it be if you could start with a simpler-to-grok all-in-one smallish application and evolve it from there, using the tools, to something increasingly more separated by concern and domain … where after some months (or years?) of working your project from a small little flower in the field to a giant rose garden, the tools have taken you down that path.

You can already do this on your own, but it is effort and you leave the tools behind as you go. To umbrella or not is a question we ask people to make at project creation time, and that’s probably not the best moment for most projects or people to make that decision. To separate data storage/retrieval from API access is a good idea, but it makes a hell of a lot more sense when you don’t have a small app (where it honestly makes very little sense: complexity for little realized benefit besides feeling good about your phyrric deisgn win) but now have something bigger. Understanding where the cut lines for smaller, focused applications (“microservices” or whatever other buzzwords one prefers ;)) is often easier for people to understand as they explore the problem space by doing.

Having tools that could help set up projects for later refactoring (and I do think Phoenix 1.3’s contexts could be a first step in that) and then help you automate those steps when they make sense for you could be absolutely empowering. It would be a way to take people on a 5 year journey of discovery, and pick for that given day in that given project where to land.

Put more simply: I don’t think we can realistically start most people down a 5 year path to glory. But we should be trying to lead them there. We can certainly start them somewhere simple, and if the tools are there for project refactoring they can accidentally stumble down the 5 year path.

15 Likes

I wonder if there’s a market for a scaffold/generator task or plug-in in mix that would delegate the last mile to generators in each lib/framework, so that @chrismccord does not need to write Ecto code inside Phoenix, leaving it to generators in Ecto.

I really resonate with this, and I feel that starting to learn Elixir/Phoenix has helped me realize that I want to be on this journey.

Big :thumbsup: for this approach!

1 Like

I guess I sort of don’t like how this is framed. Elixir is not championing them, They are just using best practices.

Technically these things are not new and are available in many languages some of which I am sure does it even better.

Umbrellas are just a term which comes from (I think) on best practices how to develop many OTP applications in the same project.

Contexts - I can’t see this being anything other than giving good old fashioned module separation[1] a new name, perhaps to help and clarify . This is the way software has been develop in many languages over long time, probably with the exception of web frameworks (php | rails | django) which evolved from simpler scripts into the web world. I don’t know why the phoenix developers decided to give them a name.

When you say you want to capitalize on them it seems more from a marketing perspective and on boarding new developers. And it looks like a classic:

  • Take old concepts and ideas
  • Rename them
  • Market them as a new and novel idea.

As demonstrated by real world these things work but from my point of view one potential problem with this is that it may appear dishonest unless it is very clear from where the ideas come.

  • You should really try smalltalk, lisp and haskell if you haven’t already. They can change the way you look at software and will help you as a developer regardless which technology you end up with).

[1] = (i.e. modules should export as little as possible and have a clearly defined API, don’t leak implementation detail from API, the module should be loosely coupled and highly coherent)

2 Likes

Watch the first several minutes of my ElixirConf 2017 keynote where I address this

1 Like

I agree with your entire post and it is well balanced describing the various trade-offs.

Perhaps this is where we should look to improve. Rather than have to make big decisions up front. Which I agree are very hard in most software. I have myself searched for best practices on how to layout code in many languages.

If there was a way to simply put it anywhere you like and then being able to automatically refactor into best practices that would be pretty cool.

On the other hand. If your blessed way is pushing people in the wrong direction you will have trouble down the line. The separate web UI / business app is the right way in the erlang world. The question is if this translates well to the way traditional web apps are made and what developers expectation of them are.

In the end @josevalim said the phoenix team has carefully weighed the trade-offs and this is all pretty new. They’ve done pretty well so far and has been willing to change direction before so give this some time and I am sure it will incrementally get better and better

1 Like

I think part of Dave’s point is that some of what he is suggesting is actually incredibly easy in Elixir because Elixir was so well designed to begin with :003:

For instance, it is super simple to create separate apps (microservices?) and then include them as dependencies. In Ruby you would need to copy and paste the entire code, or create a gem or engine or include each file of the program independently (messy) - in Elixir it is easy as adding a line to your mix.exs file.

Having said that, I don’t yet have a complete idea of what Dave’s alternative would be for everything as a whole (or even in terms of not including ecto with Phoenix - which is why I posted the thread - to learn from you wonderful people) because sadly Dave’s course does not include Ecto/a database (and I haven’t actually finished the course yet).

I think to move this discussion forward it would be incredibly valuable to see example real-world systems built with these different approaches in mind (in Elixir of course) - and then determine how accessible each of those approaches were and whether it would be worthwhile to adopt any of those ideas (as has already been done with Umbrellas, Contexts etc) :slight_smile:

In Ruby this is kind of handled by Bundler. Or documentation. In Rust it’s Cargo. Etc.

That’s exactly what Elixir is doing; whole-heartedly supporting/encouraging use of those best practices :003:

I don’t think anyone is trying to do the latter here. In fact you often hear Elixir folk sing Erlang’s praises :slight_smile:

I will… one day :lol:

With regards to naming and renaming things, I think that can be fine if it helps more easily explain something in the context of whatever it is being used in (no pun intended :p)