How to setup a Phoenix 1.3 app with Gulp/ npm scripts

antalvarenga · February 19, 2018, 8:19pm

So it seems the brunch project is on life support, so I’m considering using gulp (which seems like a nice alternative) to build my phoenix apps moving forward (see thread Potentially removing brunch from the Phoenix new template generator).

So how do you setup a phoenix 1.3 app with gulp having the same features as brunch?

amnu3387 · February 19, 2018, 9:06pm

Why not Webpack?

antalvarenga · February 20, 2018, 10:13am

Gulp seems simpler to configure

peerreynders · February 20, 2018, 11:04am

Brunch and Webpack are module bundlers. As such they have an opinion about the build process but by and large they are configured rather than scripted. In Webpack’s case things get a bit blurry because configurations are often assembled in a scripted manner.
Gulp is a task runner and as such sees itself as taking the place of npm scripts. So to bundle modules it still has to use a bundler plugin such as Browserify which often can be done simply in npm itself. Gulp has more advanced facilities than npm scripts for coordinating build tasks - but from what I’ve seen it isn’t really configured as much as it is being scripted by writing Gulp tasks. One issue is that often in cases where Gulp seems simpler it really isn’t necessary (Why I Left Gulp and Grunt for npm Scripts (2016-Jan-17)).

Modern JavaScript for Dinosaurs gives an overview of using npm scripts and Webpack (see also How JavaScript bundlers work).

Aside: Here is an example where simple shell scripts are used to build a simple react application with browserify, babel-cli, watch, and uglify, cssshrink, jest-cli, babel-jest, react-addons-test-utils, eslint, eslint-plugin-react, eslint-plugin-babel, and flow-bin.

Now I don’t know whether that is the case here but I think often people try to choose a build tool before they fully understand the build process. I’m not saying that npm scripts are the be all and end all but they are likely the best way to get acquainted with the build process and as such can be pushed quite a ways. Once the build process is better understood and one’s needs are much clearer it should be easier to pick the right tool, be it webpack (SurviveJS - webpack), parcel, rollup.js or whatever.

amnu3387 · February 20, 2018, 2:09pm

I never used gulp so I don’t really know, but I recently changed brunch to webpack and it was sort of straightforward following some tutorials (scss & assets the most annoying stuff - I can share my webpack.config if you want).

antalvarenga · February 22, 2018, 11:11pm

Thanks @peerreynders, @amnu3387.

I kinda took on @peerreynders advice and gave a try on npm scripts. I created a new phoenix project with brunch and then removed the brunch-config.js file and brunch dependencies (seems to be the most used approach to change the build tool).

Everything seems to be working, except for the js part. The built app.js file (on priv/static) on a “normal” phoenix project has a bunch of code coming from require.js, module-definitions.js, phoenix.js, phoenix_html.js, socket.js. Right now my built app.js has only the “import “phoenix_html”” statement (which is not recognized) and the socket.js code (Here is the repo of the phoenix app).

How should I deal with the javascript files, so that i can have all the default functionality that comes with phoenix?

OvermindDL1 · February 22, 2018, 11:16pm

@antalvarenda You aren’t processing your javascript through babel(?) or so, probably forgot that step?

EDIT: Yep, you are only running it through a minimizer but aren’t processing the ES6 to ES5 via whatever it was called.

antalvarenga · February 23, 2018, 12:40am

I’m more on the side of learning rather than forgetting things
Changed the build pipeline to run babel instead of uglify (i don’t want it minified on dev mode anyway).
Same result (though not minified anymore) .
app.js

peerreynders · February 23, 2018, 1:26am

Try this

$ npm i -D babel-cli babel-env-preset

add build_test/assets/.babelrc

{
  "presets": [
    ["env", {
      "targets": {
        "browsers": ["last 2 Chrome versions"]
      }
    }]
  ]
}

add these to scripts of the package.json:

"babel": "babel js/ -d build/",
"browserify": "browserify ./build/socket.js ./build/app.js ../deps/phoenix_html/priv/static/phoenix_html.js ../deps/phoenix/priv/static/phoenix.js -o ../priv/static/js/app.js",
"build:js": "npm run lint && npm run babel && npm run browserify",

For source maps you have to work a little bit harder:

// assets/build.js
const path = require('path');
const fs = require('fs');
const babelify = require('babelify');
const exorcist = require('exorcist');
const browserify = require('browserify');

const babelrc = {
  "presets": [
    ["env", {
      "targets": {
        "browsers": ["last 2 Chrome versions"]
      }
    }]
  ]
};

const config = {
  ignore: ['deps/**','node_modules/**']
};

const resolve = segment => path.resolve(__dirname, segment);

const target = fs.createWriteStream(resolve('../priv/static/js/app.js'));
const mapPath = resolve('../priv/static/js/app.js.map');

browserify({debug: true}) // turn on source maps
  .add(resolve('./js/app.js'))
  .add(resolve('./js/socket.js'))
  .add(resolve('../deps/phoenix_html/priv/static/phoenix_html.js'))
  .add(resolve('../deps/phoenix/priv/static/phoenix.js'))
  .transform(babelify.configure(config), babelrc)
  .bundle()
  .pipe(exorcist(mapPath))
  .pipe(target);

$ node build.js runs browserify, babelify and exorcist (to move the inline source map into a separate file).

amnu3387 · February 23, 2018, 10:51am

@peerreynders but at this point, when you introduce browserify it’s basically similar to using scripts&webpack right?

(this is great stuff though, thanks for sharing)

peerreynders · February 23, 2018, 1:18pm

My impression was that one of the primary reasons for using Brunch in Phoenix was to have support for (ES) modules right out of the box as it allows for cleaner frontend dependency management. With that in mind you are going to need a bundler of some description.

It’s my personal preference to use ECMAScript rather than CommonJS/Node.js style JavaScript which bundlers manage so that makes Babel a requirement.

When sticking to a “npm script based” approach it’s important to start out just using the CLI features, with a file based approach.

There is usually a point where some features aren’t directly accessible via the CLI and you have to use the JS API to get the desired results - it’s probably at this point that some people start reaching for grunt/gulp. However I’ve found that the JS API documentation for many npm packages (plug-ins) is often lacking. For example with babelify it became necessary to dive into the index.js to find out how it used babel-core to determine that the ignore configuration actually supported “A glob, regex, or mixed array of both” rather than just a single regex.

This is probably the point where it becomes a bit easier to accept dealing with the Webpack even through it may seem incredibly complex. Browserify’s API is rather restrictive. Other packages like browserify-directory can help but at this point it starts to feel like you are creating some sort of Frankenstein. Webpack has a well developed ecosystem, so there are plenty of loaders and there is usually little reason to also reach for grunt/gulp (supporting npm scripts are usually enough). Again with Webpack it is important that you start small (SurviveJS). But now having experienced all the bits and pieces that go into a JS module toolchain it should be a bit easier to understand what Webpack is trying to accomplish.

The transition to Webpack isn’t mandatory. Having a basic understanding of the aspects of the JS module tool chain it should be much easier to transition to any other tool like Rollup.js or parcel. I think the biggest hurdle for beginners with Brunch was that they had no concept of what it was doing (because they had never used JS modules before) so they had no basis from which to understand the configuration process.

Trying to use something as restricted as Browserify exposes a lot of the moving pieces which helps to build your conceptual model of the build process. That is why I’m a bit sceptical with “zero config” claims like the one’s I hear for parcel. These things are usually “zero config” until they’re not. But because so far you’ve done “zero config” you have no concept or idea where to start when it comes time to do “some config”.

antalvarenga · February 23, 2018, 1:23pm

Wow great stuff, thanks so much. Yeah I was killing myself to find a way to output the sourcemap file, should have come here sooner! I’ll try it out.

peerreynders · February 23, 2018, 1:26pm

Let us know if the source maps work (don’t have Postgres installed right now, so couldn’t fire up your project). It may be simpler to remove exorcist and leave the source map in line.

antalvarenga · February 23, 2018, 1:32pm

Yeah inline works fine…But at some point it should be in its own file, right?

peerreynders · February 23, 2018, 1:36pm

Separating them is useful for production. When they are separate the browser should be able to detect the source map but there is aways room for hiccups.

The last line of app.js is

//# sourceMappingURL=app.js.map

sourceMappingURL and sourceURL syntax changed

amnu3387 · February 23, 2018, 2:08pm

Yes - probably learning the “lower level” of calling directly the build tools from the CLI (& respective configs) would have been (and probably for me still warrants trying to learn it) a better option that jumping immediately from brunch to webpack - I will check this further thanks.

(having said that, webpack & its loaders do so much heavy lifting I don’t even feel like going to the gym anymore - sourcemaps, separate bundles, css/scss dependencies/requirements from JS, fonts, image compression, dev/prod separation in config… I’m hoping that the next iteration will also output cups of coffee on demand)

antalvarenga · February 23, 2018, 3:58pm

Yep this seems to work fine! Thanks.
One thing - here:

const config = {
  ignore: ['deps/**','node_modules/**']
};

Shouldn’t it be '../deps/**' ?

peerreynders · February 23, 2018, 4:42pm

There is a problem - but that’s not the solution. In your build.js throw in

const minimatch = require('minimatch');

other packages (i.e. babel-core) already use it so it should already be there; then throw in

console.log(
  minimatch(
    resolve('./node_modules/lodash/lodash.js'),
    'node_modules/**'
  )
);

console.log(
  minimatch(
    resolve('./node_modules/lodash/lodash.js'),
    '**/node_modules/**'
  )
);

console.log(
  minimatch(
    resolve('../deps/phoenix_html/priv/static/phoenix_html.js'),
    '../deps/**'
  )
);

console.log(
  minimatch(
    resolve('./js/socket.js'),
    '**/deps/**'
    )
);

console.log(
  minimatch(
    resolve('../deps/phoenix_html/priv/static/phoenix_html.js'),
    '**/deps/**'
  )
);

and see what happens when you $ node build.js

So it should have been ['**/deps/**','**/node_modules/**']