What would you think about a new web-framework that extends Phoenix with rails-like or django-like built-in features?

YES - easy to configure generators could have a big impact. For example: look at the complexity of installing LiveView. Right now, install-by-hand is the only option. Writing a generator for LV installation would not be easy to write or maintain, nor would a LV-install generator be easily customizable in the field.

The only generator alternative I’m aware of are PragDave’s project generator. (An Elixir Project Generator). I do use the PragDave generator, in conjunction with a set of bash scripts - it’s devolved into a spagetti-mess that is hard to maintain and impossible to share.

It’s a bit difficult to articulate exactly what the properties of a flexible/composable generator should be. Maybe you could start by enumerating use cases:

  • installing LV in a phoenix project
  • installing an authentication system in a phoenix project
  • replacing Milligram with Bootstrap4 (or some other CSS framework) in a phoenix project
  • finding the right generator for my needs
  • customizing and re-publishing a generator
  • etc.

Probably you could list desirable properties of a flexible generator:

  • creating new project from scratch, or modifying an existing project
  • supports both ‘bash’ (or exs) style scripting and template generation
  • ability to nest and/or ‘subclass’ generators
  • generators separated from their underlying application code
  • simple to share & reuse generators
  • works with both ‘umbrella’ and ‘poncho’ style layouts
  • idempotency
  • testable
  • interoperable with mix
  • etc.

One approach might be to borrow concepts from Ansible:

  • a library of prebuilt generator functions like template, line_in_file and ensure_path
  • playbooks as elixir modules or some sort of ExUnit like DSL
  • collections of playbooks published as HEX packages

It would be great to have a chunk of generator code that looked like:

generator_context
|> phx.new()
|> live_view.install()
|> auth_system.install()
7 Likes

Love the of compostable generators. I can see myself using them a lot.
Bulding new generators on top of existing generators. This sounds cool.

I agree and disagree with this at the same time. “Rolling your own” is more about creating your own primitives or cryptography schemes, not about creating your own authentication implementation, especially with library like Comeonin, which makes it truly hard to misuse (especially with the new API). In general security related APIs should be as simple as possible, but at the same time hard to misuse. libsodium is such library and for me Comeonin is also such library, it has only 3 public methods right now:

  • add_hash/1 which takes hash and return map with proper keys
  • check_pass/2 which takes struct (or nil) and provided password and check if password is valid if user exist, if not it will mock check by running check that will never pass

I do not know. If there is none, then I will happily write one. Phoenix makes it fairly easy to write authorisation implementation without getting in the way. I would say that people misusing Guardian do more harm than if they would implement auth on their own. Problems with Guardian are:

  • Using it for website (not API) and storing token in the Authentication header makes it easier to store that token incorrectly (commonly in LocalStorage) which makes it exploitable via XSS, Cookies are much more secure solution and for unknown reason people claim that this is “better”
  • JWT is terrible solution for sessions
  • JOSE is bad design

That is exactly what I am saying. Tuning existing libraries in most cases requires at least the same amount of time as tinkering your own solution. And in such case why bother in using and learning library while you can do it on your own? You just provided example why not use these libraries.

But Plug + Comeonin already provides everything needed:

  • Comeonin allows for super simple password management
  • Plug.Session for whole session management

Whole rest (welcome emails, non-password credentials, authorisation, etc.) is often highly application specific, and as I showed above, some of these are quite easy to achieve right now without many libraries.

I would say, that we would need more documentation how to do things right, not more libraries that want to do it for you because other have them. Be better than them, popular kids do not always do things right.

So one by one:

  • Authentication - as I showed, we already have all blocks there, no need for more libraries, we just need documentation how to stick all of these together
  • Authorisation - again, it can be done in a simple way. There are nice libraries that provide sugar for that, but in the end it can be simply implemented without much cruft (as showed above)
  • File uploads - just use S3-likes, I have been working in applications that thought it is better to handle file uploads through the application and I guarantee that in 99% it will fail at some point of time; it wrongfully encourages people to do file processing during upload, causes needless traffic, encourages bad practices (storing files in local FS, processing files on upload, etc.)

The problem I see is that I highly doubt that Phoenix team will want to maintain yet another library and making external libraries “blessed” is harmful for the community as it discourages people from creating alternatives. It would be perfectly reasonable (and I would encourage community to do so) to create their own, custom, generators. There could be even tool for making it easier and simpler.

Remember that LV is not even in alpha release, so nothing surprising that there is no generator/easy way to install it yet.

9 Likes

Oh. Thanks for clarifying and apologies for misunderstanding. I agree completely.

That’s kind of my point though. Same as “LISP gives you list functions, you can do everything else by yourself” is not helpful, that’s not very helpful as well. But I think we are talking past each other.

I believe mine and OP’s point is more along the lines of “we would like some such common functionality be injectable in a project without us doing much about it”. I am talking some commoditisation of things like authentication, translation, file uploads etc.

IMO it is one of the natural next paths for languages like Elixir – they should start commoditising common workflows and projects, a la Magento and WordPress. I am not talking marketing or making a language more popular – I am talking the language and ecosystem being more useful in the generic boring sense of web development.

That particular point I 100% agree with. Rolling your own modules that do simple pattern matching to assert on security privilege levels is actually much clearer compared to installing libraries. And you demonstrated it quite well.

As for JWT and the like, I always ignored them and they looked like a solution looking for a problem to me.

Agreed on the premise but I am pretty convinced that a guy on a schedule will absolutely miss a detail and that will contribute to yet another widespread private info data leak few years down the line. So even though I enjoy writing my own stuff much more, these days I turn into a conservative boring adult who prefers to use a well-made 3rd party tool and just tweak it whenever needed – even if it ends up consuming more time and effort. That way I’ll know that (1) now it does exactly what I want (covered with integration tests) and (2) I won’t contribute to the worldwide problem of being a security-incompetent dev. I rather protect my customers’ data!

That’s completely correct. :slight_smile: More docs and guides would be welcome in all common use-cases of websites which we both named on several occasions. And I also fully agree that popular rarely means good.


In the end though, some of us are looking for ways to get the boring stuff out of the way as quick as possible. I don’t get any fulfilment out of carefully thinking about all possible states of an authentication system or file uploads. This stuff should just be figured out once and for all, described in a meta-programming universal language (state machines? flow diagrams?) and written in literal stone and we should all collectively move on to doing something better with our time. :smiley:

Hence, the yearning of some devs in this community – me included – for generators that can rid us of part of this menial work.

P.S. I’d never think of trying to impose more work on the Elixir, Phoenix and Ecto maintainers. If they all quit today they’d still have done 100x more than many expected them to! :heart:

9 Likes

I think a family of pluggable packages under an overarching org would increase the trust and reliability of the packages, while keeping everything modular and lightweight.

For e.g. it would be cool to just simply add forum functionality (like firestorm) with a simple line addition to the mix deps.

5 Likes

Another option would be to create a well-documented sample Phoenix application with all of those pieces built in.

People could use the sample app as:

  • an example to learn from
  • a starting point to build new apps on top of
  • a source of code that can be copied
  • a source of git commits that can be cherry-picked
  • a git remote whose commits can be merged in to an app

Building and maintaining a sample app seems much easier than building and maintaining a framework or advanced generators, and is ultimately flexible because users can make their own decisions easily while still being guided by collective knowledge and assisted by the efficiency of having existing code to copy or learn from.

7 Likes

Yes LV is new - my comment is not specifically about LV, but more broadly about the complexity in our current generator tooling. It would be great to have scriptable generator code like:

generator_context
|> phx.new(options)
|> liveview.install()
|> my_friends_custom_css.install()
|> my_liveview_tweaks.apply()

I wouldn’t say that current generator tooling is in bad shape, writing custom Mix tasks is fairly easy and built in templating engine is quite powerful. There are tools like exgen, but these do not seem to be popular.

I agree that it would be nice to have, but I do not think that this is possible at all.

I agree. They way I see it, having things like auth and admin built on, or at least available as a flag when starting a new project, is about:

  1. Reliving the end developer of menial grunt work that doesn’t add business value.
  2. Increasig security by incorporating a standard solution that is updated and reviewed by a community, not a single developer.
5 Likes

Totally agree. It’s in that spirit I started this thread. And I’m very glad to see really nice ideas are showing up here. ^^

1 Like

I wrote up a draft README which outlines possible ideas for a new code generator. Feedback welcome.

8 Likes

I love the idea of doing what ansible does but not ansible (I have in fact twice written ansible replacement, once each for two jobs, going to open source the second one). One of the more powerful aspects of ansible is it’s idempotency. So you can run an ansible script (if it’s well written; they never are) and if it fails Midway because some tweak needs to be made, you can fix that and rerun the script and it will ignore steps that don’t need to be rerun. You may want to consider adding that aspect in, though I don’t know how necessary it is for code generation for elixir projects (if you screw up you can just rimraf the directory in many cases)

1 Like

I’ve been building a Phoenix app over the last year and have been developing in Ruby/Rails for a good 6 years. I can’t say I’ve missed anything from Rails.

Any gaps can be easily bridged with libraries and it’s straightforward to roll your own stuff. I prefer the simplicity of Phoenix and the last thing I would want is more cognitive overhead.

I agree above RE a template project. Even then you have to take into account it will be opinionated about what structure and libraries to use (another argument against baking it into Phoenix; I want the choice). You could take it a step further and develop something similar to https://github.com/RailsApps/rails-composer.

The user chooses what modules they want, whether to pull in particular libraries, and the composer builds the template structure based on that.

6 Likes

Let me know when you start to implement this I’m interested in helping if I can.

Also have some more ideas, I’ll raise a PR on the readme when I’ve thought it through

1 Like

@chrisjowen - I had another look at this yesterday, renamed the repo to UberGen, updated the README and added a test branch with some exploratory code. At this point, UberGen playbooks run end-to-end, borrowing a lot of ideas from Mix.Task and Plug.

To make this generally useful, I now believe we would need a robust refactoring library. (with functions like rename_project, extract_variable, etc.) I haven’t been able to find such tooling, and don’t have any experience with Elixir AST manipulation. Perhaps someone knows of a refactoring library, or perhaps there is an AST expert who would like to get involved.

Your PRs and suggestions are welcome!

4 Likes

Great I’ll look at this at the weekend :blush:

Some thoughts, just looking at the readme I think that what you are calling playbooks would be more close to ansible tasks. The playbook metaphore from ansible would be more like the pipeline example you gave unless I’m reading this wrong?

I like the idea of been able to link to a gist but in reality would this be enough. Where would all the template and project files live?

I can see the prompt for overwrite? but I’m thinking you would need more generic prompts for example let’s say we want to add mongo to the project, would this remove the default ecto/pg or provide both I guess that’s user decisions and the task reacts accordingly.

One final thing, while this is primarily for elixir I could see this been fantastic for other projects. I wonder if you shouldn’t segment the elixir specific things like add_mix_dependency to their own plugin/driver whatever.

1 Like

Maybe you’re right. Key distinction I think is that UberGen playbooks and helper functions can be used in nested Plug-like pipelines. Ansible playbooks and tasks can’t be composed in nested hierarchies AFAIK.

With Ansible you’ve got Playbooks, Roles, Tasks and Helper Functions (called Modules in Ansible parlance). With UberGen you’ve only got Playbooks and Helper Functions. UberGen Playbooks and Helper Functions both have a Plug-like interface ((context, options) -> new_context) . UberGen Playbooks can have dependencies, can be called from Mix command, and can have associated files and templates.

Under the priv/playbooks directory.

Probably we would want a CLI-menu function to prompt for user input.

I can see that. Will try to organize the helper functions and playbooks into domain-specific modules. To get things started I’ll focus on a single Elixir use case - adding LiveView to a Phoenix project.

As a developer advocate for security I can’t agree more with you with this.

Please NEVER roll your own Authentications and Authorization, because its never simple as you may think. Even the packages that handle them, and are battled test, are found to have bugs and security issues at some point in their lifetime.

This bugs and security issues come up to light of the day because they are used by hundreds or thousands of projects.

Now I ask for any developer to pause and think for a while… Do you really think that you can roll your own solution, that covers all corners and edge cases that Authentication and Authorization presents to the most skilled group of developers?

Well if you reply is Yes, then you must be a unicorn among developers, not one of that famous 10x developer, but a 100x one.

No intention to offend anyone, just want to bring you back to Earth :wink:

1 Like

While I partially understand authentication (but still I would dispute where “rolling your own” starts) that I completely do not understand why not authorisation. If you are talking about protocol like OAuth then I agree, but if this is authorisation within application, then why not? It is fairly easy, and if you are brave you could even use Prolog for defining rules and it will be pretty easy.

4 Likes

In OWASP API Security Top Ten, the first on the list, API1 is related with Authorization, followed by API5, and this is the ones that mention directly Authorization as a fix for them.

While Authorization is not directly mentioned as a fix for API3 and API7 issues, in my opinion a proper Authorization mechanism could be part of the fix.

So as anyone can see Authorization is a big issue when it comes to security, but guess what Authentication is in the podium to, its in the second position, as API2.

But don’t take my word for it, just start following infosec in Twitter and Linkedin, and you will see that you have lot of data breaches, that have roots on Authentication/Authorization issues.

Please always treat Security as a First Class Citizen in your development process, and not take it for granted, just because you use the framework or programming language xzy.

Laws are coming in place with huge fines for companies that suffer data breaches, like GDPR in Europe, but as Uncle Bob said once, in a near future the laws will start targeting the Developers, and I strongly believe that day will come, because we don’t care enough to self regulate ourselves, thus laws will try to do it for us, and guess what… they will also fail, but then programming will not be a fun profession anymore.

3 Likes