Potentially removing brunch from the Phoenix new template generator

[OvermindDL1]

I’m curious about peoples thoughts about this, especially of @chrismccord.

In my project I have brunch and 2 other javascript build systems as well as a dozen libraries I specifically call, it is… unwieldy.

I started looking at other popular javascript build systems recently (as shown by a past recent thread). Grunt was horrible, don’t touch it, Gulp is good, kind of like Brunch (it has watcher, reloaders, hot-reloaders, etc…), it is completely async (if the supporting libraries are, like brunch, which elm’s brunch plugin is not for one example), it is a bit more verbose but not overly so, however it is significantly more capable, looks like it can handle all the things that brunch cannot that I’ve needed. Webpack can kind of do the same things as gulp, though it takes significantly more code to do things except the most simple of cases, which ‘can’ be shorter than gulp, but not really, I’m not seeing the point in it.

However, I ran across an article: https://www.keithcirkel.co.uk/how-to-use-npm-as-a-build-tool/
I completely forgot about another build tool that comes with phoenix, npm itself!

With some simple npm packages I got watchers, reloaders, concurrent compiling, can use all the base packages that the 50-thousand gulp/webpack/brunch plugins use under the hood but directly, etc…

So I spent this morning disabling brunch just commented its like in my dev.exs file, it ‘works’, mostly (it has bugs that necessitate two-recompiles often), and rewriting everything I did in brunch and the other systems in just npm itself, following that article, the result is that it is shorter, compiles faster (mostly due to being able to run the stuff the other systems absolutely had to do synchronously but concurrently, and no multiple rebuilds), and I say it is more readable.

So… why not just use npm for the build system of the assets in Phoenix? Replace in the dev.exs template this:

node: ["node_modules/brunch/bin/brunch", "watch", "--stdin", cd: Path.expand("../", __DIR__)]

To become this:

node: ["npm", "run", "watch", cd: Path.expand("../", __DIR__)]

That will call the ‘watch’ task, which in my package.json (you can define tasks outside of the package.json in your own *.js files if you want too) is defined as:

    "watch": "parallelshell \"npm run watch:javascript\" \"'npm run watch:scss\" \"npm run watch:webcomponents\" \"npm run watch:elm\" \"npm run watch:bucklescript\" \"npm run watch:copy\"'",

And those all run concurrently, and inside those are build systems that also run concurrently. It is actually surprisingly nice. I think that I’ll run like this for a while and maybe entirely remove all the other build stuff here later.

And yes, it is *SO*FREAKING*NICE* that I do not have to double-save my files just to get brunch to notice an update (with 50 seconds in-between because the elm-brunch plugin sucks and freezes everything while it spends its 50 seconds to compile the elm stuff), and the important stuff (I.E. not the 50-second long elm compiles) are compiled about instantly so my iteration time just dropped like a rock in a vacuum high-gravity environment. ^.^

So yeah, with this change and with the recent symlink windows fix for phoenix (run as admin in dev mode…) iteration time has become fantastic again! ^.^

But yes, any thoughts as to removing brunch from the phoenix new template generator entirely and just using npm itself? It reduces dependencies, its faster, its shorter, and its less confusing then the odd brunch config (which though nice once you understand it, is definitely confusing at first).

[chrismccord]

I’m not sure it’s enough to just run a bunch of parallel node process. Ie, what about final concatenation? Script precedence/ordering? Script dependencies? Different js/css bundles? module loaders? Etc? In general, if I could escape the js ecosystem entirely and not have to include any asset builder I would, but it is an unfortunate reality. I don’t think the proposed solution covers enough or would be extensible enough for folks.

[peerreynders]

@OvermindDL1

However, I ran across an article: How to Use npm as a Build Tool - Keith Cirkel
I completely forgot about another build tool that comes with phoenix, npm itself!

An excellent article. I would also imagine that npm scripts would be a very effective way of communicating most of the information needed when somebody wants to use their favorite build tool instead - assuming that means they actually bothered to learn how to use it, rather than always relying on pre-digested configurations that are churned out by other applications.

Why I Left Gulp and Grunt for npm Scripts
Why npm Scripts?

[chrismccord]

@peerreynders

I would also imagine that npm scripts would be a very effective way of communicating most of the information needed when somebody wants to use their favorite build tool instead

This is already easily achievable today by swapping out a single line in your dev.ex :watcher configuration. For example, swapping brunch for webpack is maybe a 30s affair today.

[peerreynders]

@chrismccord

I don’t think the proposed solution covers enough or would be extensible enough for folks.

I think that you simply can’t predict all or even most of the “extension scenarios” that people may want - cover the basics then there comes a point where they need to be responsible for customizing their own environment for their own needs - and at that point it is more important that the general information on the critical aspects of the build process are readily apparent so that it can be mapped over to the desired build tool.

@chrismccord

This is already easily achievable today by swapping out a single line in your dev.ex :watcher configuration. For example, swapping brunch for webpack is maybe a 30s affair today.

I was thinking more along the lines of more fundamental details like expected/critical source and target file locations etc. I expect that right now some of this information could be buried in “the (default) way brunch works” - i.e. one would have to familiarize oneself first with brunch before employing one’s own build tool - I expect that npm scripts could expose that type of information in a more transparent way.

As it is, over the past two years I have been noticing a growing number of people declaring to have (re)discovered npm scripts either standalone or in combination with some higher level build tool (e.g. webpack).

[OvermindDL1]

@chrismccord

I’m not sure it’s enough to just run a bunch of parallel node process. Ie, what about final concatenation? Script precedence/ordering? Script dependencies? Different js/css bundles? module loaders? Etc? In general, if I could escape the js ecosystem entirely and not have to include any asset builder I would, but it is an unfortunate reality. I don’t think the proposed solution covers enough or would be extensible enough for folks.

All described in the article I linked and npm handles fine.

In parts:

@chrismccord

what about final concatenation?

Handled by the various things, like brunch uses something like browserify internally to do that, why not just use browserify straight? So I do, it handle concatenation, it uses babel to do transpiling, I can pass in my own files or even raw source into it without touching the file system, etc…

@chrismccord

Script precedence/ordering?

Also handled by browserify. Browserify itself did not exist when brunch was built, but it is basically a command-line and node library that does the same for javascript work. There are similar things for css (stylus), templates (vulcanize/crisper), webcomponents(polymer-build) , etc… (brunch’s webcomponent support is… extremely lacking, it is one of the two brunch plugins I have that require repeated recompiles because brunch does not know about them). Just call browserify straight (either the cli version or import it and use it from your npm scripts).

@chrismccord

Script dependencies?

Again, browserify.

@chrismccord

Different js/css bundles?

Specify exactly what files you want in your browserify output files, spin up an instance per output file you have, pass in the files, globs, raw source that you want it to handle.
Same with stylus for css, pass through a postcss or scss transpiler first or whatever you want too.

@chrismccord

module loaders?

Browserify is also a module loader, in addition it will also auto-polyfill things that the browser does not support that you use (Object.assign for example), but only if you use them (don’t use them? it does not include them unless you override it), brunch does not, have to load your polyfills manually.

@chrismccord

Etc? In general, if I could escape the js ecosystem entirely and not have to include any asset builder I would, but it is an unfortunate reality. I don’t think the proposed solution covers enough or would be extensible enough for folks.

The npm method can handle anything that any other javascript build system can do (you can even call into build systems if you want, that is what I was originally doing, calling brunch watch from my npm run watch until I finished moving it over) and it is the most extensible method (unlike brunch, like holy-hell-it-is-limited), while still being more succinct than any other build system I’ve seen yet, just have to not be afraid of javascript scripts. ^.^

But yeah, this would reduce the npm dependencies downloaded, it is a more simple script structure (that can even call into a build system if anyone want, or just replace it with something else by changing/adding phoenix’ watcher). The reason I suggest it instead of brunch for the default scripts (could keep a new-brunch script or something if you want, or new-gulp or new-whatever) is it is simple, fast, fewer dependencies, and explicit. It is basically a makefile with a javascript syntax. ^.^ Easier all around though.

@peerreynders

An excellent article. I would also imagine that npm scripts would be a very effective way of communicating most of the information needed when somebody wants to use their favorite build tool instead - assuming that means they actually bothered to learn how to use it, rather than always relying on pre-digested configurations that are churned out by other applications.

Why I Left Gulp and Grunt for npm Scripts
Why npm Scripts?

Ooo, more links, nice!

@chrismccord

This is already easily achievable today by swapping out a single line in your dev.ex :watcher configuration. For example, swapping brunch for webpack is maybe a 30s affair today.

Well, unless you want to have webpack do what brunch is currently doing to the project, then you have to make the webpack config file. ^.^

With npm as an initial hook someone could just call their webpack/gulp/whatever-of-choice from the npm build script itself until, if they wanted, they could replace it entirely with whatever-build-system-of-choice and then change the ‘watcher’ argument in phoenix.

I have multiple watchers in my phoenix config for note (well, ‘had’, currently just have one now since I’m testing npm run watch).

[chrismccord]

The goal is an out of the box experience that Just Works without fanfare. With a basic brunch or web pack config, a user does not have to know about browserify, how to configure it, how to bundle, etc. So even if this can all be achieveved by stringing fewer deps together, end-users still have to know how those deps work individually and configure to their liking. This may work well for power users but newcomers, especially those not enthralled in js day to day, just want to compile and bundle js/es20xx/sass without pain.

[alexgaribay]

I don’t find any issue with Brunch being the default configured Node asset bundler/transpiler. For me, I prefer Webpack since a do a lot of React development. However, I suspect most developers don’t care or don’t know all the nuances of choosing a Node asset bundler and just prefer to start with some that just works. Like Chris mentioned, it doesn’t take very long to replace Brunch in your application. Additionally, EEX templates are pretty awesome and fast .

[OvermindDL1]

@chrismccord

just want to compile and bundle js/es20xx/sass without pain.

Problem is that brunch is indeed painful, from teaching it to quite a few people as well as remembering my initial experience with it that was obvious.

To do the JS/es20** part is just browserify, its command-line is simple, like here is mine for my app.js:

"build:app.js": "browserify web/static/js/app/**/*.js -e web/static/js/app/index.js -o priv/static/js/app.js -t [ babelify --presets [ es2015 ] ]",

I don’t use a vendor directory but that can be added easily enough to browserify with another parameter to not have it transform those files. If you notice the same options that brunch scatters around in (a couple initially confusing) places are just right there, and you can have unique options for files too (want to run the ‘react’ babelify preset on only react files, you can do that easily here, not at all easily in brunch, if at all without making your own plugin).

For css I just have (since I use scss):

"build:app.css": "node-sass -o web/static/css/app.css web/static/css/app.scss",

And of course the "build:outputfilename" is just a convention I’ve been using, can use whatever you want.

@chrismccord

With a basic brunch or web pack config, a user does not have to know about browserify, how to configure it, how to bundle, etc.

But they do have to know about brunch/webpack, which is significantly larger of a thing to learn. I started by “Hmm, how do I output two javascript file where I can import between them”, which in brunch ended up being a couple of lines of specifying path regex’s and such, simple once I saw, but no clue how to do that at first and took a bit of research as the first couple of tries caused brunch to start throwing errors.

Where in npm, everyone knows the shell (considering calling npm/mix/etc… uses it).

@alexgaribay

However, I suspect most developers don’t care or don’t know all the nuances of choosing a Node asset bundler and just prefer to start with some that just works.

Which brings up my issue, they do not ‘just work’. It works for the single JS file and single CSS file that phoenix includes, but those are trivial cases. When I expanded beyond that brunch became hell to work with. As stated, even now I have to save my file, wait 50 seconds, save my file again (or just ‘touch’ brunch-config.js to have the entire brunch watcher reload) to get the files to update properly.

I enable brunch again, I make an edit in my JS file that I have open, I save it, the phoenix page reloads…and does not include the change. I save it again, wait a bit, page reloads again and still no change, I touch brunch-config.js, wait a couple of minutes, the webpage reloads about 8 times while it compiles, then finally the change is seen. Then I notice a small bug and my element is not being cleared from the DOM, so I make the fix, save it, the page reloads while I switch to the terminal and hit then again to run touch brunch-config.js, wait a couple of minutes, and it is fixed, but now I have to make the next section…

I was dreading every-single-edit that I had to make. Maybe it is a Windows thing, but brunch is painful and its time has been increasing substantially. The NPM replacement I made this morning is significantly shorter and I’ve not had a single issue of having to recompile a file yet, plus a full rebuild now only takes the time that the longest task takes (which is elm compiling) instead of longer.

[alexgaribay]

@OvermindDL1

Which brings up my issue, they do not ‘just work’. It works for the single JS file and single CSS file that phoenix includes, but those are trivial cases. When I expanded beyond that brunch became hell to work with. As stated, even now I have to save my file, wait 50 seconds, save my file again (or just ‘touch’ brunch-config.js to have the entire brunch watcher reload) to get the files to update properly.

Anecdotally, I use multiple SCSS files and multiple JS files and I’ve never had issues with the transpile time. Brunch transpiles my code in a matter of seconds but I develop on OS X instead of Windows, albeit I just export a single CSS and JS file when all is finished.

[andre1sk]

I think we can come up with some nice package to do what you propose vs making it default. So those who need it would just do --no-brunch and install a package that has a task to init everything to the way you describe.

[OvermindDL1]

@alexgaribay

Anecdotally, I use multiple SCSS files and multiple JS files and I’ve never had issues with the transpile time. Brunch transpiles my code in a matter of seconds but I develop on OS X instead of Windows, albeit I just export a single CSS and JS file when all is finished.

Yeah I’ve not had these issues on my linux desktops at home, though my projects there are quite a bit smaller and less complex then the one at work. So the one at work, where I have all the issue, is both far far larger and it is on Windows, either one could be why brunch is painful there, either one is bad for having brunch as a default in phoenix though.

@andre1sk

I think we can come up with some nice package to do what you propose vs making it default. So those who need it would just do --no-brunch and install a package that has a task to init everything to the way you describe.

Honestly I’d say default should be the npm way, with a --brunch to enable brunch, --webpack to enable webpack, etc… The default way should be the most simple and reliable, and brunch is definitely not reliable on all platforms that Phoenix supports. And debugging into webpack/gulp issues is remarkably painful (that is something that brunch does better than them).

[chrismccord]

I think the node world is painful no matter what approach you take and there’s no silver bullet. My main concern with the proposed approach is you end up accidentally rebuilding brunch, but worse, with a hodgepodge of dependent scripts. I think it’s a great choice if it’s working well for you, but I remain unconvinced that it’s the best approach for the majority of users.

[andre1sk]

@OvermindDL1

Honestly I’d say default should be the npm way, with a --brunch to enable brunch, --webpack to enable webpack, etc… The default way should be the most simple and reliable, and brunch is definitely not reliable on all platforms that Phoenix supports. And debugging into webpack/gulp issues is remarkably painful (that is something that brunch does better than them).

I am guessing there is a fairly low probability of that change so having a convenient package might be the second best option . and having it merged into phoenx so --webpack to enable webpack --npm to just use npm --gulp to use gulp would be nice.

[OvermindDL1]

@chrismccord

I think the node world is painful no matter what approach you take and there’s no silver bullet.

Hear hear, that is why I’m trying to remove as much of it as I can.

@chrismccord

My main concern with the proposed approach is you end up accidentally rebuilding brunch, but worse, with a hodgepodge of dependent scripts. I think it’s a great choice if it’s working well for you, but I remain unconvinced that it’s the best approach for the majority of users.

It is a way that many npm projects do things I’ve been noticing today. It is not that you are rebuilding brunch, rather you are just calling the main interfaces that things like brunch try to hide from here (thus causing the plugins to be out of date, missing features, etc… etc…) from a shell-like interface (which can include calling a custom script via node myScript if you want, but I’ve not actually had to do that surprisingly). Each npm shell entry in packages.json is just a single shell command each, with it documented that &&, <, >, and | will be supported on all shells on all platforms that the scripts run on, so keeping to that, using things like rimraf (like my clean command is "clean": "rimraf priv/static/*",) instead of rm -rf for multi-platform, then you are good and it works and you can easily tear things apart or run commands yourself to see exactly what they do. No need for a multitude of plugins in whateverBuildSystem that may or may not be up to date (many in brunch are not), may or may not support the features you may need from the underlying library the plugin uses (many of my brunch plugins had not, hence why I’ve had to rewrite a few of them), when all is exposed fine through the normal node interface. Like here was my elm plugin section in brunch-config.js:

    elmBrunch: {
      elmFolder: '.',
      mainModules: [
        'web/elm/MessengerApp.elm',
        'web/elm/NotificationApp.elm'
      ],
      // If these next two are specified, all mainModules will be compiled to a single file (optional and merged with outputFolder)
      outputFolder: 'web/static/js',
      outputFile: 'elm.js',
      makeParameters: ['--warn', '--yes']
    },

The npm version is:

"build:elm.js": "elm make web/elm/*.elm --output web/static/js/elm.js --warn --yes"

Elm cannot output to stdout, but nor can the brunch plugin or anything, and I was just converting the brunch style. I’m thinking of changing the output style a bit.

I still find it funny that the elm make command takes over 40 seconds to compile (find web/elm/ -iname '*.elm' | wc -l) 31 files, yet bsb.exe (bucklescripts compiler) takes < 1.5s to build (find web/bucklescript -iname '*.ml' | wc -l) 55 files and elm is (via du) is 190K (and compiles to 891K javascript, ~730K after minimization) and bucklescripts is 908k (I’ve built a very large library, compiles to 1.1megs javascript, ~280K after minimization, BS uses the javascript module system along with a design that is able to be optimized well, also it puts in a lot of comments, where elm output has no comments)).

@andre1sk

I am guessing there is a fairly low probability of that change so having a convenient package might be the second best option . and having it merged into phoenx so --webpack to enable webpack --npm to just use npm --gulp to use gulp would be nice.

I know, I’m just trying to spur conversation.

Currently it is mostly just ‘it should be easy’ but there has been no examples of talk on ‘how’ brunch/webpack is easier than npm, and currently I’m finding it to be the opposite, as well as faster.

[josevalim]

@OvermindDL1

But yes, any thoughts as to removing brunch from the phoenix new template generator entirely and just using npm itself? It reduces dependencies, its faster, its shorter, and its less confusing then the odd brunch config (which though nice once you understand it, is definitely confusing at first).

If we had replaced the assets engine every time someone found an allegedly new/better/faster solution, we would be on our 26th interaction with possibly no or little noticeable improvement. If brunch is not for you, then go and replace it by something else. We chose brunch for everyone who doesn’t want to care or cares very little about the js build tool.

This is the second or third time this week we are on the same discussion. It does not help. We have limited time and I would prefer to use that time in not rehashing topics that have already been debated in such short time span.

[sasajuric]

@josevalim

We chose brunch for everyone who doesn’t want to care or cares very little about the js build tool.

Just to back this claim, at my company we care little about js build tool, and brunch works perfectly for us. It provides a low ceremony out of the box experience that does the job gets out of the way. So I’m personally really happy with that. My understanding is that more demanding users can easily replace brunch with anything else, so I’m not sure why this topic is raised so frequently?

[josevalim]

@josevalim

It does not help.

To be a bit less negative and contribute more positively to the discussion: what would help is to document those steps in articles, blog posts and get people trying it out. The article someone shared a couple days ago about Webpack with Phoenix was great (and it was the first time I could understand how to configure Webpack).

[andre1sk]

I don’t think the issue is with demanding users , while not being an advocate of change I think the frequency of issue being raised still might be a good indicator that there might be a need for some solution.

[OvermindDL1]

@josevalim

We chose brunch for everyone who doesn’t want to care or cares very little about the js build tool.

That is what I’ve been curious about, it builds very poorly, no support of other processes, webcomponents, image processing (spritesheet generation, minimizing/optimizing), etc… It essentially has support for javascript with babel, css (and for things like scss you have to choose to either output css or js, cannot have both), and an unprocessed asset path that is copied verbatim (no ability to have something like svg icons in node_modules somewhere auto-update, svg-minimize, and output an output svg file + css file). The big issue however is how it randomly and often misses files as the project gets bigger when running on Windows (which is what started all of this), and that is even with just javascript.

Do feel free to ignore these though, it does work fine on at least linux for the simple purpose it is included for if not a bit verbosely.

@sasajuric

Just to back this claim, at my company we care little about js build tool, and brunch works perfectly for us. It provides a low ceremony out of the box experience that does the job gets out of the way. So I’m personally really happy with that. My understanding is that more demanding users can easily replace brunch with anything else, so I’m not sure why this topic is raised so frequently?

It breaks often, and can be replaced with a single line (or two if you want watch support) in the npm package.json file, which will not break. It provides a bad user experience that I’ve had to walk a few people through.

@josevalim

To be a bit less negative and contribute more positively to the discussion: what would help is to document those steps in articles, blog posts and get people trying it out. The article someone shared a couple days ago about Webpack with Phoenix was great (and it was the first time I could understand how to configure Webpack).

Heh, I actually used that article in my webpack testing (older than a few days? Or a different one), but it ends up being even more verbose than gulp/brunch in its setup and configuration. I am thinking of putting up something that documents it, would not be hard (need to make a new project, my structure is so changed up because certain javascript libraries are stupid and need specific things in specific places to build…).

@andre1sk

I don’t think the issue is with demanding users , while not being an advocate of change I think the frequency of issue being raised still might be a good indicator that there might be a need for some solution.

I’ve not noticed others so it might be my fault? Sorry for the bugging. ^.^

But yeah, translating brunch to npm scripts should be easy, will try to find the time this weekend to document it up (my work is forcing me to stop working from home on weekends and instead use my free time as free time because of new laws going into effect in the USA, blehg, wasted time is what it is…).