Help debugging edeliver Phoenix site not showing

Continuing on from another recent post of mine where I was helped to fix a Phoenix 1.3 app use edeliver to build it (thank you), I’m now finding I can start the production server, but can’t see the app.

Here’s what I’ve done/checked:

  • It runs locally on my mac on port 4000
  • I’m building and hosting it on the same server that has erlang installed
  • Just to be sure, I’ve tried building it with and without the erlang runtime

It’s behind an nginx reverse proxy server that hasn’t changed in settings since previously successfully built/deployed/ran a Phoenix 1.2 app on the same server.

I’m trying to understand how to read the logs in /var/log/ to see if the site is working.

Where do I start?

What is it you can’t see? Phoenix generated content or static files? What do you see instead? Can you SSH into your server and verify that your application id running, listening and serving via ps, netsat, curl/links/lynx?

  • testing this with the default phx.new project - it can’t be seen.

  • the nginx reverse proxy server redirects from http://www.mydomain to http://mydomain and gives a 404.

  • the public ip of the build/production server - returns nothing.

Here’s the top output

      PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME    WCPU COMMAND
     9100 freebsd       1  20    0 20120K  3668K RUN      0:09   0.35% top
    10057 freebsd      11  20    0  1638M 65088K select   0:19   0.23% beam
    10220 postgres      1  20    0   197M 27140K select   0:01   0.03% postgres
     8986 freebsd       1  20    0 83092K  7884K select   0:02   0.03% sshd
    10222 postgres      1  20    0   197M 24368K select   0:01   0.01% postgres
    10227 postgres      1  20    0   197M 24380K select   0:01   0.01% postgres
    10224 postgres      1  20    0   197M 24368K select   0:01   0.01% postgres
    10225 postgres      1  20    0   197M 24368K select   0:01   0.01% postgres
    10232 postgres      1  20    0   197M 24380K select   0:01   0.01% postgres
    10229 postgres      1  20    0   197M 24380K select   0:01   0.01% postgres
    10230 postgres      1  20    0   197M 24380K select   0:01   0.01% postgres
    10233 postgres      1  20    0   197M 24380K select   0:01   0.01% postgres
    10234 postgres      1  20    0   197M 24380K select   0:01   0.01% postgres
    10221 postgres      1  20    0   197M 24368K select   0:01   0.01% postgres
    10228 postgres      1  20    0   197M 24380K select   0:01   0.01% postgres
    10226 postgres      1  20    0   197M 24380K select   0:01   0.01% postgres
    10223 postgres      1  20    0   197M 24368K select   0:01   0.01% postgres
    10231 postgres      1  20    0   197M 24380K select   0:01   0.01% postgres
      655 postgres      1  20    0   195M 22416K select   0:11   0.00% postgres
      661 postgres      1  20    0 54276K 12140K select   0:08   0.00% postgres
      660 postgres      1  20    0   195M 23164K select   0:07   0.00% postgres
      659 postgres      1  20    0   195M 22472K select   0:02   0.00% postgres
      658 postgres      1  20    0   195M 22420K select   0:01   0.00% postgres
     1324 freebsd       1  20    0 14840K  2540K select   0:01   0.00% epmd
      915 root          1  20    0 12564K  2424K nanslp   0:01   0.00% cron
      466 root          1  20    0 10472K  2396K select   0:00   0.00% syslogd
      626 _ntp          1  20  -20 14244K  3216K select   0:00   0.00% ntpd
      391 root          1  20    0  9512K  4960K select   0:00   0.00% devd
      657 postgres      1  20    0   195M 22484K select   0:00   0.00% postgres
      649 root          1  38    0 55676K  7024K select   0:00   0.00% sshd
      910 smmsp         1  20    0 20600K  5956K pause    0:00   0.00% sendmail
     8984 root          1  21    0 83092K  7796K select   0:00   0.00% sshd
     8987 freebsd       1  20    0  6480K  2852K wait     0:00   0.00% bash
    10054 freebsd       1  20    0 14812K  2452K select   0:00   0.00% run_erl
      628 root          1  20  -20 14236K  3084K select   0:00   0.00% ntpd
    10218 freebsd       1  31    0 14832K  2344K select   0:00   0.00% inet_gethost
      627 _ntp          1  48    0 14236K  3148K select   0:00   0.00% ntpd
    10150 freebsd       1  31    0  8300K  1936K select   0:00   0.00% erl_child_setup
      982 root          1  52    0 10460K  2080K ttyin    0:00   0.00% getty
      975 root          1  52    0 10460K  2084K ttyin    0:00   0.00% getty
      977 root          1  52    0 10460K  2084K ttyin    0:00   0.00% getty
    10219 freebsd       1  29    0 14832K  2452K piperd   0:00   0.00% inet_gethost
      980 root          1  52    0 10460K  2084K ttyin    0:00   0.00% getty
      979 root          1  52    0 10460K  2084K ttyin    0:00   0.00% getty
      981 root          1  52    0 10460K  2084K ttyin    0:00   0.00% getty
      974 root          1  52    0 10460K  2084K ttyin    0:00   0.00% getty
      978 root          1  52    0 10460K  2084K ttyin    0:00   0.00% getty
      976 root          1  52    0 10460K  2084K ttyin    0:00   0.00% getty
     5935 root          1  52    0  6244K  1876K nanslp   0:00   0.00% sleep
     5931 root          1  52    0 13144K  2684K wait     0:00   0.00% sh

Is there a cowboy log or something I can check to see if the site is being served?

Are you using Distillery to build a release? If so, do you have server: true set in your prod.exs?

Yeah, and for whatever reason that line was uncommented - I’ve uncommented it now thanks.

However, I’m getting a 502 error after the redirect from www to the domain - here’s the exact error from the reverse proxy nginx error.log:

*1 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: xxx.xx.xx.xx, server: mytestphoenixapp.tld, request: "GET / HTTP/1.1", upstream: "http://xx.xxx.xx.xxx:8080/", host: "mytestphoenixapp.tld"

I explicitly set the http port in prod.exs instead of relying on the default as it says one should (in Phoenix 1.3):

config :mytestphoenixapp, MytestphoenixappWeb.Endpoint,
  http: [port: 8080],
  url: [host: "mytestphoenixapp.tld", port: 80],
  cache_static_manifest: "priv/static/cache_manifest.json"

Is this just a path issue in my nginx config?

Are you able to curl localhost from the server itself? If you SSH into the machine and curl localhost:8080 what happens?

On the production server:

curl localhost:8080

gives:

curl: (7) Couldn't connect to server

which is confusing because from the dev mac I can:

mix edeliver start production

returns:

START DONE!

and

mix edeliver ping production

returns:

pong

Did you ever get this working? It sounds like the server isn’t listening on port 8080 for some reason. Can you post your redacted prod.exs file?

I think it is the same problem as in another thread where I asked him for the prod.exs yesterday…

Hi jesse,

Thanks - no, I didn’t get it going - you’re right the server appears to not be listening on port 8080, and I couldn’t find any reason for that. The strange thing is, the Phoenix 1.2 app I had working with Distillery and Edeliver uses the same port, nginx reverse proxy config…

To try to find out what’s wrong, I’ve decided to start from scratch with the default Phoenix 1.3 install, and see if you and others can help me spot what I’m doing wrong.

  • The app is called phx1p3.

  • I want to build the app for deployment with erts included, not install erts on the production server.

  • My production server user for the deployed app is called freebsd.

On dev machine


Install Phx 1.3 App

$ mix phx.new phx1p3

Fetch and install dependencies? [Yn] y
* running mix deps.get
* running mix deps.compile
* running cd assets && npm install && node node_modules/brunch/bin/brunch build

$ cd phx1p3


Generate new secret for app:

$ mix phx.gen.secret

Copy outputted secret into config/prod.secret.exs


Add elixir/phoenix packages

to mix.exs for what my project needs eg bamboo for email:

$ vim mix.exs

Add:

{:bamboo, "~> 0.8.0"},
{:bamboo_smtp, "~> 1.2"},
{:secure_random, "~> 0.5.0"},
{:comeonin, "~> 2.6"},
{:edeliver, "~> 1.4.3"},
{:distillery, "~> 1.4.1"}

Install NPM packages:

$ cd phxp1p3/assets

$ npm install --save-dev copycat-brunch

$ npm install --save font-awesome

Update assets/brunch-config.js for the installed NPM packages:

$ vim assets/brunch-config.js

Change:

// Configure your plugins
  plugins: {
    babel: {
      // Do not use ES6 compiler in vendor code
      ignore: [/vendor/]
    }
  },

to:

// Configure your plugins
  plugins: {
    babel: {
  // Do not use ES6 compiler in vendor code
  ignore: [/vendor/]
    },
    copycat: {
      fonts: ["node_modules/font-awesome/fonts"]
    }
}

Install Phoenix and NPM packages/dependencies:

$ mix deps.get —all

Create project database:

$ mix ecto.create

Test app works locally:

$ mix phx.server


Git

$ git init

Add Edeliver’s releases directory to .gitignore

$ echo ".deliver/releases/" >> .gitignore

$ git add .

$ git commit -m "initial commit"


Edit prod.exs

Replace load_from_system_env: true, with http: [port: 8080], and uncomment the line config :phoenix, :serve_endpoints, true.

$ vim config/prod.exs

use Mix.Config

# For production, we often load configuration from external
# sources, such as your system environment. For this reason,
# you won't find the :http configuration below, but set inside
# Phx1p3Web.Endpoint.init/2 when load_from_system_env is
# true. Any dynamic configuration should be done there.
#
# Don't forget to configure the url host to something meaningful,
# Phoenix uses this information when generating URLs.
#
# Finally, we also include the path to a cache manifest
# containing the digested version of static files. This
# manifest is generated by the mix phx.digest task
# which you typically run after static files are built.
config :phx1p3, Phx1p3Web.Endpoint,
#  load_from_system_env: true,
  http: [port: 8080],
  url: [host: "myapp.tld", port: 80],
  cache_static_manifest: "priv/static/cache_manifest.json"

# Do not print debug messages in production
config :logger, level: :info

# ## SSL Support
#
# To get SSL working, you will need to add the `https` key
# to the previous section and set your `:url` port to 443:
#
#     config :phx1p3, Phx1p3Web.Endpoint,
#       ...
#       url: [host: "example.com", port: 443],
#       https: [:inet6,
#               port: 443,
#               keyfile: System.get_env("SOME_APP_SSL_KEY_PATH"),
#               certfile: System.get_env("SOME_APP_SSL_CERT_PATH")]
#
# Where those two env variables return an absolute path to
# the key and cert in disk or a relative path inside priv,
# for example "priv/ssl/server.key".
#
# We also recommend setting `force_ssl`, ensuring no data is
# ever sent via http, always redirecting to https:
#
#     config :phx1p3, Phx1p3Web.Endpoint,
#       force_ssl: [hsts: true]
#
# Check `Plug.SSL` for all available options in `force_ssl`.

# ## Using releases
#
# If you are doing OTP releases, you need to instruct Phoenix
# to start the server for all endpoints:
#
     config :phoenix, :serve_endpoints, true
#
# Alternatively, you can configure exactly which server to
# start per endpoint:
#
#     config :phx1p3, Phx1p3Web.Endpoint, server: true
#

# Finally import the config/prod.secret.exs
# which should be versioned separately.
import_config "prod.secret.exs"

Install Distillery and Edeliver

Edeliver

Create directory and config file for Edeliver:

$ mkdir .deliver

$ touch .deliver/config

$ vim .deliver/config

Paste and edit config from Deploy Early and Often: Deploying Phoenix with Edeliver and Distillery (Part Two), and change:

  • prefix phoenix. to phx. (for Phoenix 1.3).

  • add change of path for change in Phoenix 1.3 directory eg cd '$BUILD_AT/assets' for NPM install.

      APP="phx1p3"
    
      BUILD_HOST="xxx.xx.xx.xx"
      BUILD_USER="freebsd"
      BUILD_AT="/tmp/edeliver/$APP/builds"
    
      #RELEASE_DIR="/tmp/edeliver/$APP/builds/_build/prod/rel/$APP"
    
      # prevent re-installing node modules; this defaults to "."
      GIT_CLEAN_PATHS="_build rel priv/static"
    
      STAGING_HOSTS="xxx.xx.xxx.xxx"
      STAGING_USER="freebsd"
      TEST_AT="/home/freebsd/staging"
    
      PRODUCTION_HOSTS="xxx.xx.xx.xx"
      PRODUCTION_USER="freebsd"
      DELIVER_TO="/home/freebsd/www"
    
      # For *Phoenix* projects, symlink prod.secret.exs to our tmp source
      pre_erlang_get_and_update_deps() {
        local _prod_secret_path="/home/freebsd/prod.secret.exs"
        if [ "$TARGET_MIX_ENV" = "prod" ]; then
          __sync_remote "
            ln -sfn '$_prod_secret_path' '$BUILD_AT/config/prod.secret.exs'
          "
        fi
      }
    
      pre_erlang_clean_compile() {
        status "Running phx.digest" # log output prepended with "----->"
        __sync_remote " # runs the commands on the build host
          # [ -f ~/.profile ] && source ~/.profile # load profile (optional)
          source ~/.profile
          # echo \$PATH # check if rbenv is in the path
          set -e # fail if any command fails (recommended)
          cd '$BUILD_AT' # enter the build directory on the build host (required)
          # prepare something
          mkdir -p priv/static # required by the phx.digest task
          cd '$BUILD_AT/assets'
          npm install
    
          ./node_modules/brunch/bin/brunch build --production
    
          # run your custom task
          cd '$BUILD_AT'
          APP='$APP' MIX_ENV='$TARGET_MIX_ENV' $MIX_CMD phx.digest $SILENCE
        "
      }
    

Distillery

Create Distillery rel directory and config file:

$ mix release.init

Edit the generated Distillery config file to set:

  • default_environment: :prod

  • dev environment to not build erts

  • prod environment to build erts

$ vim rel/config.exs

# Import all plugins from `rel/plugins`
# They can then be used by adding `plugin MyPlugin` to
# either an environment, or release definition, where
# `MyPlugin` is the name of the plugin module.
Path.join(["rel", "plugins", "*.exs"])
|> Path.wildcard()
|> Enum.map(&Code.eval_file(&1))

use Mix.Releases.Config,
    # This sets the default release built by `mix release`
    default_release: :default,
    # This sets the default environment used by `mix release`
    default_environment: :prod

# For a full list of config options for both releases
# and environments, visit https://hexdocs.pm/distillery/configuration.html


# You may define one or more environments in this file,
# an environment's settings will override those of a release
# when building in that environment, this combination of release
# and environment configuration is called a profile

environment :dev do
  # If you are running Phoenix, you should make sure that
  # server: true is set and the code reloader is disabled,
  # even in dev mode.
  # It is recommended that you build with MIX_ENV=prod and pass
  # the --env flag to Distillery explicitly if you want to use
  # dev mode.
  set dev_mode: false
  set include_erts: false
  set cookie: :"@ts,~1]ld]%|CJkB1El@r$mgpcWloZ$kK15JnxJAJ!{XuwwBRQp/%iAV4qW?Ggse"
end

environment :prod do
  set include_erts: true
  set include_src: false
  set cookie: :"|P[:(]X=SI_bRRm7[V,$=_UKFneJW3$Q[z*BrfyjmR..`lTIFao($R;*RRcLXgPQ"
end

# You may define one or more releases in this file.
# If you have not set a default release, or selected one
# when running `mix release`, the first release in the file
# will be used by default

release :phx1p3 do
  set version: current_version(:phx1p3)
  set applications: [
    :runtime_tools
  ]
end

On Staging or Production Server

Create prod.secret.exs

in deployment user’s home directory:

$ touch ~/prod.secret.exs

Copy contents of config/prod.secret.exs on dev machine to the above file on the server via clipboard.

Create Postgresql Database and Assign User Privileges

Switch to Postgresql user

$ sudo -i -u postgres

Log into Postgresql console

$ psql

postgres=# CREATE DATABASE phx1p3_prod;

Create user with same name as the user account running the Phoenix app:

postgres=# CREATE USER freebsd WITH PASSWORD 'password';

Grant that user privileges to new database

postgres=# GRANT ALL PRIVILEGES ON DATABASE phx1p3_prod TO freebsd;

Quit Postgresql and exit user to server’s deployment user:

postgres=# \q

$ exit

Build, Deploy, and Start on Production Server

$ mix edeliver build release production --verbose

$ mix edeliver deploy release to production --verbose

$ mix edeliver start production

Nginx Server Config

Relevant part of nginx.conf:

$ cat /usr/local/etc/nginx/nginx.conf

#Production Server hosting Phx1p3 app

upstream phx1p3 {
   server xx.xxx.xx.xxx:8080;
}

#Nginx Reverse Proxy Server's redirect request to www.myapp.tld; on port 80 to http:/myapp.tld; on port 80

    server {
        listen       80;
        server_name  www.myapp.tld;
        return 301 $scheme://myapp.tld$request_uri;
}

    server {
        listen       80;
        server_name  myapp.tld;

        location /{
    	try_files $uri @proxy;
        }

        location @proxy {
        # include proxy_params;
        proxy_redirect off;
        proxy_pass http://phx1p3;
        }
 }
1 Like

Again:

How does your prod.exs look like? After starting your application, what does netstat -lnpt tell you (maybe that its not available on BSD)?

Thanks Nobbz, I should have included the config/prod.exs, sorry. I’ve updated my latest post with it. However, as noted in an earlier post, uncommenting

config :phoenix, :serve_endpoints, true

or

config :phx1p3, Phx1p3Web.Endpoint, server: true

makes no difference.

I’ll get back to you with netstat -lnpt or the equivalent soon.

I only skimmed edelivers documentation, but whats the value of your MIX_ENV?

https://github.com/bitwalker/distillery/blob/master/docs/Common%20Issues.md#why-do-i-have-to-set-both-mix_env-and---env

Go on your server and try to connect to your port from dev.exs via curl, lynx, wget, whatever. If that works, try to rebuild and redeploy using MIX_ENV=prod.

Instead of just uncommenting those lines, try putting it in the config block like so instead.

config :phx1p3, Phx1p3Web.Endpoint,
#  load_from_system_env: true,
  http: [port: 8080],
  server: true,
  url: [host: "myapp.tld", port: 80],
  cache_static_manifest: "priv/static/cache_manifest.json"

Thanks jesse,

I’ve just tried that now and I’m getting this error - a new one that I haven’t seen before. I’m feeling jinxed!

mix edeliver build release production --verbose

-----> Compiling sources
Checking whether deps must be compiled for mix version 1.3.[01234]
Compiling 13 files (.ex)
Generated phx1p3 app
-----> Generating release
using mix to generate release
==> Loading configuration..
==> Assembling release..
==> Release failed, unable to load selected release
    - Make sure `rel/config.exs` has at least one release configured
    - Make sure at least one is set as default OR
    - Pass --name=<rel_name> to `mix release`

Have you run mix release.init yet?

Hey jesse,

Yeah, I ran it and it created the config.exs… which didn’t work the first time, so I’ve re initialised distillery with

$ mix release.init

and I’m getting the same error.

:frowning:

Can you post your rel/config.exs?

Thanks jesse,

$ cat rel/config.exs

# Import all plugins from `rel/plugins`
# They can then be used by adding `plugin MyPlugin` to
# either an environment, or release definition, where
# `MyPlugin` is the name of the plugin module.
Path.join(["rel", "plugins", "*.exs"])
|> Path.wildcard()
|> Enum.map(&Code.eval_file(&1))

use Mix.Releases.Config,
    # This sets the default release built by `mix release`
    default_release: :default,
    # This sets the default environment used by `mix release`
    default_environment: Mix.env()

# For a full list of config options for both releases
# and environments, visit https://hexdocs.pm/distillery/configuration.html


# You may define one or more environments in this file,
# an environment's settings will override those of a release
# when building in that environment, this combination of release
# and environment configuration is called a profile

environment :dev do
  # If you are running Phoenix, you should make sure that
  # server: true is set and the code reloader is disabled,
  # even in dev mode.
  # It is recommended that you build with MIX_ENV=prod and pass
  # the --env flag to Distillery explicitly if you want to use
  # dev mode.
  set dev_mode: false
  set include_erts: false
  set cookie: :"86$eKxz@y<XV8$)S,)o}J|nXq((%U8r@H@>[_@ikkbP?e9OD^q03l`;31kIYa>0Q"
end

environment :prod do
  set include_erts: true
  set include_src: false
  set cookie: :"*|8%iyp/*a=8GZx~IT:7qT_f<e0MIhHym4*U80{z{xT~vOLZaVwGTaR.FB;uXTac"
end

# You may define one or more releases in this file.
# If you have not set a default release, or selected one
# when running `mix release`, the first release in the file
# will be used by default

release :phx1p3 do
  set version: current_version(:phx1p3)
  set applications: [
    :runtime_tools
  ]
end

I’ve also tried setting the default release (as per my config posted earlier) with the following line changed

from:

default_environment: Mix.env()

to:

default_environment: :prod

What happens if you remove the plugin section?