Flexible Dockerized Phoenix Deployments (1.2 & 1.3)

Nope, I’m on Debian using the testing branch :wink:

It seems to be actually my fault. My umask is on 0077 which produces the files with 600 permission.

1 Like

Id like to use https://github.com/SteveLTN/https-portal as the nginx-server, which will allow me to get an ssl cert from letsencrypt.

my changes to the docker-compose.yml in step 8(I’m using localhost for local testing):

version: "3"
services:
  server:
    image: steveltn/https-portal:1
    container_name: nginx-server
    ports:
      - '80:80'
      - '443:443'
    environment:
      STAGE: local
      DOMAINS: "olivetree"
    volumes:
      - ./conf.d:/etc/nginx/conf.d:ro
    networks:
      - nginx-network
networks:
  nginx-network:
    external: true

and in conf.d:

server {
    listen 443;
    server_name localhost;
    location / {
        proxy_pass http://olivetree-server:5000/;
        proxy_set_header Host $host;
        proxy_buffering off;
    }
}

but I get a couple errors that don’t really make sense to me:

nginx-server | /opt/certs_manager/lib/nginx.rb:11:in `initialize': Read-only file system @ rb_sysopen - /etc/nginx/conf.d/olivetree.conf (Errno::EROFS)

and

nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: bind() to 0.0.0.0:443 failed (98: Address already in use)

I don’t understand why it’s trying to write to the conf file and above all, why the address is already in use.

Any ideas how to get past this all? Is there a better way to get updating ssl certs and https support?

full error log:

```
•100% ➜ docker-compose up  
Creating nginx-server ... done
Attaching to nginx-server
nginx-server | [fix-attrs.d] applying owners & permissions fixes...
nginx-server | [fix-attrs.d] 00-runscripts: applying... 
nginx-server | [fix-attrs.d] 00-runscripts: exited 0.
nginx-server | [fix-attrs.d] done.
nginx-server | [cont-init.d] executing container initialization scripts...
nginx-server | [cont-init.d] 00-welcome: executing... 
nginx-server | 
nginx-server | ========================================
nginx-server | HTTPS-PORTAL v1.3.0
nginx-server | ========================================
nginx-server | 
nginx-server | [cont-init.d] 00-welcome: exited 0.
nginx-server | [cont-init.d] 10-persist-env: executing... 
nginx-server | [cont-init.d] 10-persist-env: exited 0.
nginx-server | [cont-init.d] 20-setup: executing... 
nginx-server | Generating DH parameters, 2048 bit long safe prime, generator 2
nginx-server | This is going to take a long time
nginx-server | .....................................+..........................................................+........................................................................................................................................+.............+..................................................+................................................+..................................................+..............................................................+.....................................................................................................+...............................................+.................+...................................................................................................................................................................................................................................................................+......+......................................................................................+..............................................................................................................................................................+...........................................................................................+................................................................................................+...................................+.............................+.................................................+...............................+...............+...............................+...........................................................................+...........................................................................................+....................................................................................................................................................................................................................................+...............................................................................................................+...........................................................................................+.........+.........................................+.......................................................................................................................................................+..........................................................................................................................................................+...............................................+.............................+..................................................................+................................................................................................................................................+...................................+..................................................................+.......+.....................................+...............................................................................................................................................................+...................................+...........................+................+.......................+................+.................................................+...........+................................................................................................................................+..............+..................................................................+......................................................................+.....................+.........+................................................................................................+.........................................................................................+................................................+...+............................+.......................................................................................................................+..............................+.............+............+............................+.............+.............................................................................................................................+..............................................................................+............................................................................................................................+..........................................................................+......................................................................................................................................................................+......................+.....+..+...................................+.................................................................+.........................................................................................................................................................................................................................................................................+..........................................................................................................................................+...................................+.................................................................................................................................................................................................................................................................+................................+.........................................+....+..........................................................................+....................................+..............................+...............................................................+............+............................................................................................................................................................+...............+.............................................................................+...........................................................................+.....................................+...........+..................................................................................+..................................................................+........++*++*
nginx-server | Generating RSA private key, 4096 bit long modulus
nginx-server | ................++
nginx-server | ..................................................................................................................................++
nginx-server | e is 65537 (0x010001)
nginx-server | /opt/certs_manager/lib/nginx.rb:11:in `initialize': Read-only file system @ rb_sysopen - /etc/nginx/conf.d/olivetree.conf (Errno::EROFS)
nginx-server | 	from /opt/certs_manager/lib/nginx.rb:11:in `open'
nginx-server | 	from /opt/certs_manager/lib/nginx.rb:11:in `config_http'
nginx-server | 	from /opt/certs_manager/certs_manager.rb:53:in `block (2 levels) in ensure_signed'
nginx-server | 	from /opt/certs_manager/certs_manager.rb:52:in `each'
nginx-server | 	from /opt/certs_manager/certs_manager.rb:52:in `block in ensure_signed'
nginx-server | 	from /opt/certs_manager/certs_manager.rb:77:in `block in with_lock'
nginx-server | 	from /opt/certs_manager/certs_manager.rb:75:in `open'
nginx-server | 	from /opt/certs_manager/certs_manager.rb:75:in `with_lock'
nginx-server | 	from /opt/certs_manager/certs_manager.rb:51:in `ensure_signed'
nginx-server | 	from /opt/certs_manager/certs_manager.rb:21:in `setup'
nginx-server | 	from /etc/cont-init.d/20-setup:4:in `<main>'
nginx-server | [cont-init.d] 20-setup: exited 1.
nginx-server | [cont-init.d] 30-set-docker-gen-status: executing... 
nginx-server | [cont-init.d] 30-set-docker-gen-status: exited 0.
nginx-server | [cont-init.d] done.
nginx-server | [services.d] starting services
nginx-server | [services.d] done.
nginx-server | Starting crond ...
nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx-server | nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx-server | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx-server | nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx-server | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx-server | nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx-server | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx-server | nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx-server | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx-server | nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx-server | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx-server | 2018/09/16 02:32:47 [emerg] 178#178: still could not bind()
nginx-server | nginx: [emerg] still could not bind()
nginx-server | [cont-finish.d] executing container finish scripts...
nginx-server | [cont-finish.d] done.
nginx-server | [s6-finish] syncing disks.
nginx-server | [s6-finish] sending all processes the TERM signal.
nginx-server | [s6-finish] sending all processes the KILL signal and exiting.
nginx-server exited with code 0
```

Honestly I don’t have any clue about SSL or certificates. I would try to make sure the files have the correct permissions, I don’t know why it would want to access that path. The second error looks like there’s some other kind of SSL service running for some reason.

I just got it working actually…!

Unfortunately your :ro conf files will not work with https-portal because https-portal does it’s own configuring.
I’m not sure if I can take advantage of forwarding to multiple internal phoenix instances…but not worrying about that right now. Here is the final .yml:

version: "3"
services:
  server:
    image: steveltn/https-portal:1
    container_name: nginx-server
    ports:
      - '80:80'
      - '443:443'
    environment:
      #STAGE: local
      STAGE: 'production'
      DOMAINS: "****.com -> http://olivetree-server:5000/"
    networks:
      - nginx-network
networks:
  nginx-network:
    external: true
1 Like

So I upgraded distillery from 1.5.3 to 2.0.10 and I’m getting an error in the -admin container:

olivetree-admin | Loading olivetree..
olivetree-admin | {"init terminating in do_boot",{{badmatch,{error,{already_loaded,olivetree}}},[{'Elixir.Olivetree.ReleaseTasks',seed,0,[{file,"lib/olivetree/release_tasks.ex"},{line,19}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
olivetree-admin | init terminating in do_boot ({{badmatch,{error,{already_loaded,olivetree}}},[{Elixir.Olivetree.ReleaseTasks,seed,0,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})
olivetree-admin | 
olivetree-admin | Crash dump is being written to: erl_crash.dump...done
olivetree-admin exited with code 1

Not sure why it’s telling me it is already_loaded

Here is the code in question:

I’ve also tried:

:ok = Application.load(me) // no change

log:

~/webapp/phx/olivetree (master ✘)✹ ᐅ docker logs olivetree-admin 
Loading olivetree..
{"init terminating in do_boot",{{badmatch,{error,{already_loaded,olivetree}}},[{'Elixir.Olivetree.ReleaseTasks',seed,0,[{file,"lib/olivetree/release_tasks.ex"},{line,19}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({{badmatch,{error,{already_loaded,olivetree}}},[{Elixir.Olivetree.ReleaseTasks,seed,0,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Crash dump is being written to: erl_crash.dump...done

This does not happen with distillery 1.5.3.

But with distillery 2.0.10, it happens both with 1.6.6-otp-20 AND 1.7.3-otp-21

Michael

I think there are a lot of changes in Distillery 2.0+, I haven’t looked into it very much. The only thing I can think of is that maybe now you don’t have to call Application.load/1 anymore because the app is already started for some reason?

Yes, just removing that line seems to fix it!

1 Like

So now that Phoenix 1.4 and Ecto 3.0 are out, I’m testing this build flow again. I’m running into an issue with this line in release_tasks.ex:

Ecto.Migrator.run(repo, migrations_path(repo), :up, all: true)

This is the output(linked in the comments):

It appears to have something to do with sql logging. But the strange thing is that there was a late bug fixed about this:

I’m certain I have ecto_sql@3.0, from mix.exs {:ecto_sql, "~&gt; 3.0"}, so I’m not sure what might be going on…

Any suggestions?

Michael

You need to add ecto_sql to the @start_apps list.

2 Likes

Thanks, yes that seemed to fix one issue. I have a new migration error stack unfortunately.

The top of the stack seems to be: 'Elixir.Ecto.Adapters.SQL',raise_pool_size_error

Any suggestions on how to debug this?

What a weird error message, it seems like the error message failed to print the string and instead printed out the binary form. I pasted it into IEx to get the actual message :stuck_out_tongue: Looks like you need to make sure to have the pool_size: 10 config key in your configuration.

Erlang/OTP 21 [erts-10.1.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Interactive Elixir (1.7.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> m = <<77,105,103,114,97,116,105,111,110,115,32,102,97,105,108,101,100,32,116,111,32,114,117,110,32,98,101,99,97,117,115,101,32,116,104,101,32,99,111,110,110,101,99,116,105,111,110,32,112,111,111,108,32,115,105,122,101,32,105,115,32,108,101,115,115,32,116,104,97,110,32,50,46,10,10,69,99,116,111,32,114,101,113,117,105,114,101,115,32,97,32,112,111,111,108,32,115,105,122,101,32,111,102,32,97,116,32,108,101,97,115,116,32,50,32,116,111,32,115,117,112,112,111,114,116,32,99,111,110,99,117,114,114,101,110,116,32,109,105,103,114,97,116,111,114,115,46,10,87,104,101,110,32,109,105,103,114,97,116,105,111,110,115,32,114,117,110,44,32,69,99,116,111,32,117,115,101,115,32,111,110,101,32,99,111,110,110,101,99,116,105,111,110,32,116,111,32,109,97,105,110,116,97,105,110,32,97,32,108,111,99,107,32,97,110,100,10,97,110,111,116,104,101,114,32,116,111,32,114,117,110,32,109,105,103,114,97,116,105,111,110,115,46,10,10,73,102,32,121,111,117,32,97,114,101,32,114,117,110,110,105,110,103,32,109,105,103,114,97,116,105,111,110,115,32,119,105,116,104,32,77,105,120,44,32,121,111,117,32,99,97,110,32,105,110,99,114,101,97,115,101,32,116,104,101,32,110,117,109,98,101,114,10,111,102,32,99,111,110,110,101,99,116,105,111,110,115,32,118,105,97,32,116,104,101,32,112,111,111,108,32,115,105,122,101,32,111,112,116,105,111,110,58,10,10,32,32,32,32,109,105,120,32,101,99,116,111,46,109,105,103,114,97,116,101,32,45,45,112,111,111,108,45,115,105,122,101,32,50,10,10,73,102,32,121,111,117,32,97,114,101,32,114,117,110,110,105,110,103,32,116,104,101,32,69,99,116,111,46,77,105,103,114,97,116,111,114,32,112,114,111,103,114,97,109,109,97,116,105,99,97,108,108,121,44,32,121,111,117,32,99,97,110,32,99,111,110,102,105,103,117,114,101,10,116,104,101,32,112,111,111,108,32,115,105,122,101,32,118,105,97,32,121,111,117,114,32,97,112,112,108,105,99,97,116,105,111,110,32,99,111,110,102,105,103,58,10,10,32,32,32,32,99,111,110,102,105,103,32,58,109,121,95,97,112,112,44,32,82,101,112,111,44,10,32,32,32,32,32,32,46,46,46,44,10,32,32,32,32,32,32,112,111,111,108,95,115,105,122,101,58,32,50,32,35,32,97,116,32,108,101,97,115,116,10>>
"Migrations failed to run because the connection pool size is less than 2.\n\nEcto requires a pool size of at least 2 to support concurrent migrators.\nWhen migrations run, Ecto uses one connection to maintain a lock and\nanother to run migrations.\n\nIf you are running migrations with Mix, you can increase the number\nof connections via the pool size option:\n\n    mix ecto.migrate --pool-size 2\n\nIf you are running the Ecto.Migrator programmatically, you can configure\nthe pool size via your application config:\n\n    config :my_app, Repo,\n      ...,\n      pool_size: 2 # at least\n"

P.S. @michalmuskala does this look like an Ecto bug printing error messages that way?

1 Like

Still struggling with the migration to ecto 3.

getting the error saying that it can’t find :database

olivetree-server | 
olivetree-server | 04:33:07.686 [error] GenServer #PID<0.1832.0> terminating
olivetree-server | ** (RuntimeError) connect raised KeyError exception: key :database not found.The exception details are hidden, as they may contain sensitive data such as database credentials. You may set :show_sensitive_data_on_connection_error to true if you wish to see all of the details
olivetree-server |     (elixir) lib/keyword.ex:386: Keyword.fetch!/2
olivetree-server |     (postgrex) lib/postgrex/protocol.ex:90: Postgrex.Protocol.connect/1
olivetree-server |     (db_connection) lib/db_connection/connection.ex:66: DBConnection.Connection.connect/2
olivetree-server |     (connection) lib/connection.ex:622: Connection.enter_connect/5
olivetree-server |     (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
olivetree-server | Last message: nil

How can I debug this? not sure where to set the :show_sensitive_data_on_connection_error

Also make sure to increase the pool size in your ReleaseTasks Module, see this blog post (which helped me fix the same error).
TL;DR in YourApp.ReleaseTasks search for this line:

Enum.each(@repos, & &1.start_link(pool_size: 1))

and increase pool_size to at least 2

This looks like a very interesting repo. I wonder if it is still relevant and up to date in January 2023?

I haven’t worked on Phoenix apps in a long time, so this hasn’t been updated in a while unfortunately.