Hi there, should we make the phx.new more flexible.
Let’s say we could allow the user could use their template (along with the default template) like the Rails Application Templates so the user could easy adding their setup automatically without doing it manually, for example, they would like to pre-setup Mox, Wallaby, Oban,…
Some thing like phx.new MyAwesomeApp --live --template="./template_path".
For now, I’m trying to modify phx_new by adding my extra_steps there, not sure it’s a good solution or not.
I think with the extra step hook there, it will save a lot of time, many services company can bootstrap a new project with their customization template (they could add some pre-configuration, some best practice, their convention) without setup everything manually.
That’s something I did with Rails like 7 years ago to build a bunch of custom templates. It helped a lot.
It let you build templates by writing code like:
initial_commit
update_gitignore
copy_gemfile
copy_base_favicon
add_dotenv
add_procfile
add_markdown_readme
add_license
# ... 30 other things to supply a bunch of opinionated app decisions.
And then there were tons of helper functions provided by Rails to help you inject code at specific points in files and copy files from A to B.
There’s no need for that stuff to be part of phx.new. See for example phx.gen.auth or the pow generators. They simply ship as third party packages without any need of bundling it with phoenixs installer.
The beauty of what Rails did was provide you a framework for creating your own base applications and generators with the least amount of pain.
There were dozens of helper functions built in to make changes to files (insert at line, insert after regex match, etc.), a whole DSL for stuff like this. Without this, then every person who ever creates a generator will need to re-invent the wheel.
Also having the new generator accept a single URL based template opens up a lot of great options for the community to build upon. Without this, we’re stuck having to roll our own custom unique snowflake solution to glue together a bunch of generators.
There’s already Mix.Generator at the heart of all those tools. If that’s missing critical things I guess there’s always the possibility to port generalizable functionality from existing library generators to that module or a third party library if it can’t be added to the core. But I don’t see a need for integration into phoenixs tools here. I looked at phx.gen.auth this week for creating a PR and it’s really not that much code tbh. It’s like 95% templates and the rest seems to be dealing mostly with the differences available through the config, creating certain values dynamically based on the project it’s applied to and such.
That sounds like it can be useful, but why should it be built into phoenix tooling. This is something many libraries can benefit from. I guess nerves could easily benefit from something like that.
Generator helpers and the application template are separate pieces of functionality. Right now we have a generator but it’s missing a lot of features and if you check the auth generator it has a bunch of custom functions to inject code at points in a file, but it’s all hard coded for the auth generator.
If it’s not built into Phoenix, where would it live? Having it officially built into the generator would let the Phoenix community build something like what Rails has with Rails Bytes because it wouldn’t be on us to create a non-standard DSL for generators in every generator.
Laravel also has GitHub - laravel/jetstream: Tailwind scaffolding for the Laravel framework. which is the most comprehensive generator system I’ve ever seen in any framework. It just came out too. Something like this is a killer feature of a framework for getting going quickly if your goal is to build a web app and ship a product.
As I said. I won’t deny that such generators might be a boon to setting up projects. I’d support anyone trying to improve in that space. But why integrate with phoenix at that point. Elixir is so much more than phoenix projects and I really don’t see how tooling around generating code should be limited to just the popular webframework. By your arguments why should nerves people build their own generator ecosystem, why should the ash framework build its own generator ecosystem? If Mix.Generator does miss features then that’s a chance to talk about improving it. I doubt any of the people involved in the currently scattered places of code would mind anyone trying to consolidate that work.
Yeah, make sense, we shouldn’t limit that in only the Phoenix Framework, I will look on Mix.Generator and phx_gen_auth and play around to check whether it works like what I’m looking.
Besides that, How about we adding this technique to Phoenix documentation so the other people will know how to create a template in the right way? What do you think about this idea?
I find there is much more work to be done during initial project setup than installing a few Hex dependencies and running some rote Mix tasks. We also tend to need to intrusively edit some existing files of our projects to incorporate those extensions, because it’s difficult to impossible to get magical, automatic behavior by installing a dep alone - which is a good thing. Line-edits to files that are Elixir code and not merely structured data are hard to perform in a repeatable, automated, safe way, so that usually falls to a human to complete based on reviewing documentation.
Among the steps I take when beginning a new Elixir project are: defining CI/testing, configuring coverage expectations, installing and configuring dialyxir, configuring releases and a Dockerfile, etc.[1] As those are all moderately organization-specific and don’t have a singular objective best practice in the community, I don’t think they belong in phx.new, but I can see how a bespoke set of generator logic for a given org makes a lot of sense if you’re standing up new projects frequently enough.
1: I’ve seen José tweet about a possible future Dockerfile generator sometime in the last few months. I don’t remember if it was a generalized tool or specific to the context of Phoenix, and I have no idea if that is work in progress or not.
Rails and Laravel do this in a way where the community has built very thriving ecosystems of generated code and application templates by doing line edits.
It’s doable, as long as you’re willing to accept that in some cases it may require slight human intervention to resolve conflicts if you’re pulling in sources from independent authors. But generally speaking it can be done.
As respectfully as I can, I believe this is not something to emulate and is part of why people are paid very well to maintain legacy applications in these ecosystems.
I just check the phx_gen_auth, it seems that relies on 3 components
Mix.Phoenix
Mix.Generator
EEx
I went around that a bit, it seems like copy a new file is easy but for injecting the content to the existing file, that required some manual work like the Injector module
Moving forward, I think we should create some APIs for the Mix.Generator to inject or replace the content, for example this one Mix.Generator.inject_content("mix.exs", "{:mox, "~> 1.0"},", "\n{:wallaby, "~> 2.0"}" , direction: :after) means adding the \n{:wallaby, "~> 2.0"} after the {:mox, "~> 1.0"} in the mix.exs file.
Just be aware that because such injections are not really fool-prove (it’s code we’re talking about not some documented data format) something like that might not get approved for inclusion into mix itself. There’s a reason we don’t have a mix deps.install mox. But if that’s the case a common third party library for doing that will do the job nearly as fine.
I explored this topic a while ago for my team to adopt elixir easily with all common tool installed.
At first, I wrote a bash script to find & replace and commit step by step, so I have the full history from phx.new… and it worked well, but making every step reliable (e.g. step X may require step Y… or whatever) can be tricky.
After reviewing few template tools for elixir project bootstrapping… I ended up using exgen - with my fix for supporting recent elixir versions.
For my use case, I just need a way to bootstrap elixir project (ideally with some flavors), and such template tools work well. I maintain several templates such as phoenix with ecto and without ecto. It has lots of duplicates, but anyway I can diff templates easily so pretty maintainable so far.
The rails generator methods seems useful, and I see the value of such automation (e.g. rails provides rails app:update command)… but I think that’s more feasible in rails only because rails (and its ecosystem) has many convention (e.g. more than convention over configuration). In contrast, I think Elixir tends to be more flexible in the code structure or implementation, as far as it works as expected from outside point of view. It feels like… automating wrong thing (it reminds me changing code for phpbb extension).
Without a defacto standard way (such as what Rails does with application templates and its generators), then it’s the wild west in how you implement additions to a default generated application. Rails application templates are much more than externally loading a URL. It’s what goes inside of the external template that matters and Rails has a ton of great things to make that experience amazing.
With Elixir it feels like we’re starting from scratch with nothing to use and if it’s up to the community to do it, there’s no standard way, which is going to make it really hard to get everyone on board.
Basically most languages and frameworks have a very good solution to this problem, where as we do not. And for quickly bootstrapping certain types of projects, it’s really nice to have generators and application templates.
The core team likely thinks the same because otherwise Jose wouldn’t have bothered creating the auth generator.
I see the benefits of Rails application template, but my use case is only for bootstraping customized projects, not adding some features later, and exgen works much better for me.
It would be better if you provide more concrete example
Elixir/Phoenix has less convention and more flexibility on organization than Ruby/Rails… it can be challenging. Also adding a helper at wrong abstraction to Elixir core tool would be bad so I really like the Elixir’s conservative approach so far.
(Don’t get me wrong - I wanted to have similar feature! I believe we need to discuss problems to solve, not how to solve the problem - that’s why I’m asking for your specific needs!)
Kind of related - you can see the philosophy a little bit here
Rails puts models, views and controllers in a specific directory and configuration is done in 1 spot.
Phoenix puts contexts, views, templates and controllers in a specific directory and configuration is done in 1 spot.
Of what? I think the Rails application template design is close to as good as it gets for creating such a thing. It’s all there to take inspiration from and potentially improve upon.
Yes, but also phoenix suggest quite strongly that business logic doesn’t belong into controllers and while you do have some semblance of structure in the web layer you don’t really have that for the core application at all. Sure there’s contexts, but taken with a grain of salt you can also say that’s just “have modules and functions, where the latter act as a public api”.
I think it’s more than that. The docs and generators all push for using contexts in a specific way. It would seem reasonable to use that for where you put your core business logic from a generator.
If you wanted to go off the beaten path, you probably aren’t in the market of someone who uses generators and also even if you still wanted to use them, it’s way easier to move around fully working code into different files and directories than write it all from scratch.
Please note that Rails use autoload, and expects specific file name and structure for ruby classes - while Phoenix does not do that for elixir modules.
What is a “thing”? That’s what we need to discuss to capture our needs, before making something.
For example, steps in the following list is not Elixir-specific actually and thus it doesn’t need something from Elixir.
Let me give my user stories and my decisions, for your information. It would be helpful you describe your cases in more details, other than "
I want to add oban to new or existing project for background processing, which requires probably 20 lines of changes.
Can I automate it? Mostly yes, but adding supervisor new supervisor to the main supervisor is tricky…
Will it be worth to automate it? For new project - it’s small (since files are not changed) but it’s not worth to make it work on existing projects.
I maintain a few projects, having slightly different dependencies (e.g. postgresql / oban / kafka / channel…). How can I ensure they have up-to-date convention/config easily?
Is it ever solvable problem? Unless I make another abstraction (e.g. having own library), it’s kind of writing a program to read and rewrite a program like me. I think that’s not easy right now.
We have lots of non-elixir specific common files (CI config, docker config, issue template, githook, commit lint)
Should I manage them with Elixir project generator? Maybe for some (like docker) but most are language/tool agnostic so I don’t want to manage them with Elixir generator.
Don’t get me wrong - I’m not saying such project generator or application template is meaningless. Instead, I’d like to have good ecosystem, where we have right tools at right level - such as Elixir provides easy way to manipulate code and emit formatted code, and then we can have community tools around it.
But to have right tools at the right layer, we need to know what the actual user stories are (going back to that discussion)
BTW I wish we have language-agnostic tools for language agnostic tasks. For example, we may have a language-agnostic tool to run such steps, calling language/framework specific steps - and each language provides building block so that step authors can use good APIs (may vary by language though). (oh… it reminds my previous bash script - I have one for rails and another for elixir…)
I’m just not happy to have package.json to use changelog generation (standard-version), git hooks (husky), commit message convention check (@commitlint/cli)
On exgen - @cdegroot gave a talk about using exgen in PagerDuty in 2019 to start new project ready to deploy (e.g. having all CI/CD config).
