Distillery Migrations (Win 10): configuration for MyApp.Repo not specified in :my_app environment

migrations
distillery
configuration
microsoft-windows
phoenix

#1

Hello everyone!

I’m finally breaking down and reaching out to the community for help on this topic. Over the summer, before Distillery 2.0 came out, I spent a good deal of time learning about releases and compile-vs-runtime configuration and so on, not to mention general development of a Phoenix app (first real one I’ve done, :boom: ). After discovering and making a few necessary bug fixes to the ps1 files (we’re on Windows), we got everything working pretty well. Except for migrations and seeds, that is. I spent days on it, trying everything I could think of and every suggestion from every semi-related thread or blog post I could find, but to no avail. We ended up having to manually generate sql files using ecto.dump and pg_dump. There are a couple issues even still with this approach, but it got us through development of the app.

Fast-forward to this week, Distillery 2.x is out and with it many nice improvements (thank you!), we’re done with development, and we’re ready to hand everything over to the client. I thought I’d revisit the migration/seeding issue with a fresh mindset and a new version of Distillery. After several more days, I’m stuck again, though I did get a bit further. I ran into another minor issue in how Distillery discovers custom user commands on Windows (which you’ll see in a minute) but after getting around that, I ultimately am stumped with the following error: ** (ArgumentError) configuration for PhoenixDistillery.Repo not specified in :phoenix_distillery environment.

It’s pretty clear the configuration for the Repo is not available when running this, but it’s also clearl, at least to me, that I still don’t truly understand how everything fits together, and I’m really hoping some kind and more enlightened soul out there can not only point out what I’m doing wrong, but help me understand why. In support of that, below I will provide mostly exact replication steps to get from ground zero to the exact error shown above. So here goes!

Elixir 1.6.6
Erlang/OTP 21 (compiled w/ 19)
Phoenix 1.3.3
Ecto 2.2.11
Distillery 2.012
PostgreSQL 10
IDE: VSCode running as Administrator
using regular cmd terminal, not powershell terminal

# Create Phoenix project
mix phx.new --no-html --no-brunch phoenix_distillery

# Add distillery and plug_cowboy to deps in mix.exs

defp deps do
    [
      ...,
      {:plug_cowboy, "~> 1.0"},
      {:distillery, "~> 2.0"}
    ]
  end

# Modify config/prod.exs - I’m hardcoding things here for simplicity

use Mix.Config

config :phoenix_distillery, PhoenixDistilleryWeb.Endpoint,
  http: [port: {:system, "PORT"}],
  url: [host: "localhost", port: {:system, "PORT"}],
  server: true,
  root: ".",
  version: Application.spec(:phoenix_distillery, :vsn)

config :logger, level: :info

config :phoenix_distillery, PhoenixDistilleryWeb.Endpoint,
  secret_key_base: "your_secret_key_base"

config :phoenix_distillery, PhoenixDistillery.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: "postgres",
  password: "your_password",
  database: "phoenix_distillery_prod",
  hostname: "localhost",
  pool_size: 15

# Generate a simple schema and migration
mix phx.gen.schema Blog.Post blog_posts title:string views:integer

# Add the following to priv/repo/seeds.exs

alias PhoenixDistillery.{Blog, Repo}

_post = %Blog.Post{title: "My furst poast", views: 0}
|> Repo.insert!

Create lib\release_tasks.ex

defmodule PhoenixDistillery.ReleaseTasks do
    @start_apps [:crypto,  :ssl, :postgrex,  :ecto]
    @repos Application.get_env(:phoenix_distillery, :ecto_repos, [])

    def migrate(_argv) do
        start_services()
        run_migrations()
        stop_services()
    end

    def seed(_argv) do
        start_services()
        run_migrations()
        run_seeds()
        stop_services()
    end

    defp start_services do
        IO.puts("Starting dependencies..")
        Enum.each(@start_apps, &Application.ensure_all_started/1)
        IO.puts("Starting repos..")
        Enum.each(@repos, &(&1.start_link(pool_size: 1)))
    end

    defp stop_services do
        IO.puts("Success!")
        :init.stop()
    end

    defp run_migrations do
        IO.puts("Running migrations..")
        Enum.each(@repos, &run_migrations_for/1)
    end

    defp run_migrations_for(repo) do
        app = Keyword.get(repo.config, :otp_app)
        IO.puts ("Running migrations for #{app}")
        migrations_path = priv_path_for(repo, "migrations")
        Ecto.Migrator.run(repo, migrations_path, :up, all: true)
    end

    defp run_seeds do
        Enum.each(@repos, &run_seeds_for/1)
    end

    defp run_seeds_for(repo) do
        seed_script = priv_path_for(repo, "seeds.exs")
        if File.exists?(seed_script) do
            IO.puts("Running seed script..")
            Code.eval_file(seed_script)
        end
    end

    defp priv_path_for(repo, filename) do
        app = Keyword.get(repo.config, :otp_app)
        repo_underscore =
            repo
            |> Module.split()
            |> List.last()
            |> Macro.underscore()

        priv_dir = "#{:code.priv_dir(app)}"
        Path.join([priv_dir, repo_underscore, filename])
    end
end

# Pull deps, compile, create db, init distillery
mix deps.get --only prod
set "MIX_ENV=prod" && mix compile
set "MIX_ENV=prod" && mix ecto.create
mix release.init

# Create rel\commands\win folder for migrate and seed commands
mkdir rel\commands\win

# Create migrate.ps1
Write-Host "Running migrate command..." Release-Ctl eval --mfa "PhoenixDistillery.ReleaseTasks.migrate/1" --argv -- "$args"

# Create seed.ps1
Write-Host "Running seed command..." Release-Ctl eval --mfa "PhoenixDistillery.ReleaseTasks.seed/1" --argv -- "$args"

# Update rel/config.exs - note the overlays are necessary b/c of bug in how Distillery tries to discover custom user commands on windows (phoenix_distillery.ps1 looks for commands inside _build\prod\rel\releases\<ver>\commands\win but without extra overlays it copies the .ps1 files in the same manner as the .sh files to _build\prod\rel\releases\<ver>\commands

~w(rel plugins *.exs)
|> Path.join()
|> Path.wildcard()
|> Enum.map(&Code.eval_file(&1))

use Mix.Releases.Config,
    default_release: :default,
    default_environment: Mix.env()

environment :dev do
  set dev_mode: true
  set include_erts: false
  set cookie: :"your_dev_cookie"
end

environment :prod do
  set include_erts: true
  set include_src: false
  set cookie: :"your_prod_cookie"
  set vm_args: "rel/vm.args"
end

release :phoenix_distillery do
  set version: current_version(:phoenix_distillery)
  set applications: [
    :runtime_tools
  ]
  set commands: [
    migrate: "rel/commands/win/migrate.ps1",
    seed: "rel/commands/win/seed.ps1"
  ]
  set overlays: [
    {:mkdir, "releases/<%= release_version %>/commands/win"},
    {:copy, "rel/commands/win/migrate.ps1", "releases/<%= release_version %>/commands/win/migrate.ps1"},
    {:copy, "rel/commands/win/seed.ps1", "releases/<%= release_version %>/commands/win/seed.ps1"}
  ]
end

# Build release (it’s not clear to me whether --env=prod is necessary
set "MIX_ENV=prod" && mix release --env=prod

# Try to run in foreground (should work)
set "PORT=4000" && .\_build\prod\rel\phoenix_distillery\bin\phoenix_distillery.bat foreground

# Close and try to run migration
set "COOKIE=ytho" && .\_build\prod\rel\phoenix_distillery\bin\phoenix_distillery.bat migrate

# Result

Running migrate command...
Starting dependencies..
Starting repos..
Running migrations..
** (EXIT from #PID<0.88.0>) an exception was raised:
    ** (ArgumentError) configuration for PhoenixDistillery.Repo not specified in :phoenix_distillery environment
        (ecto) lib/ecto/repo/supervisor.ex:32: Ecto.Repo.Supervisor.runtime_config/4
        (ecto) lib/ecto/repo/supervisor.ex:153: Ecto.Repo.Supervisor.init/1
        (stdlib) supervisor.erl:295: :supervisor.init/1
        (stdlib) gen_server.erl:374: :gen_server.init_it/2
        (stdlib) gen_server.erl:342: :gen_server.init_it/6
        (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

Also of note: my sys.config file in _build\prod\rel\phoenix_distillery\releases\<ver>\sys.config

[{distillery,[{config_providers,[]}]},
 {sasl,[{errlog_type,error},{sasl_error_logger,false}]},
 {logger,
     [{console,
          [{format,<<"$time $metadata[$level] $message\n">>},
           {metadata,[user_id]}]},
      {level,info}]},
 {phoenix_distillery,
     [{ecto_repos,['Elixir.PhoenixDistillery.Repo']},
      {'Elixir.PhoenixDistilleryWeb.Endpoint',
          [{render_errors,
               [{view,'Elixir.PhoenixDistilleryWeb.ErrorView'},
                {accepts,[<<"json">>]}]},
           {pubsub,
               [{name,'Elixir.PhoenixDistillery.PubSub'},
                {adapter,'Elixir.Phoenix.PubSub.PG2'}]},
           {http,[{port,{system,<<"PORT">>}}]},
           {url,[{host,<<"localhost">>},{port,{system,<<"PORT">>}}]},
           {server,true},
           {root,<<".">>},
           {version,"0.0.1"},
           {secret_key_base,
               <<"my_secret_key_base">>}]},
      {'Elixir.PhoenixDistillery.Repo',
          [{adapter,'Elixir.Ecto.Adapters.Postgres'},
           {username,<<"postgres">>},
           {password,<<"my_password">>},
           {database,<<"phoenix_distillery_prod">>},
           {hostname,<<"localhost">>},
           {pool_size,15}]}]}].

Non-exhaustive list of other things I’ve tried (and failed at perhaps b/c it isn’t intended to work that way, or perhaps because of user-error/misunderstanding :slight_smile:) :

  • leaving it running in foreground and running migrate in another terminal
  • leaving it running in foreground and running migrate in another terminal setting cookie to the same prod cookie
  • using the Mix Config Provider and copying the prod.exs file using an overlay to etc/config.exs
  • running the commands from a pre_start hook (couldn’t actually get this to work even for an echo, but didn’t spend a ton of time on it. Like commands, seems like there may be some discrepancies and needed tweaks in ps1 files as compared to sh files)
  • using REPLACE_OS_VARS=true like in Distillery 1.x days (is this obsolete now? such confuse)
  • reading DB config vars in Repo.init via System.get_env and ensuring they’re set as env variables both during release creation and during command execution
  • probably half a dozen other variations I can’t remember

Whew!

So, if you made it this far, you are already my hero. If you think you can help troubleshoot this with me, you will not only be my hero, but you will actively be participating in helping convince a .NET shop that it’s OK, and maybe even preferable!, to do a second app with Elixir :smiley: You will be having a Real Impact! Haha. But really, thank you in advance to everyone who gets this far. I love this language, I love this community (even though I’ve been in the shadows, generally speaking), and I’m like…90% certain I’m just overlooking something totally obvious to your keen eyes that I just don’t grok yet.

Thank you! :crossed_fingers:


#2

Yeesh, sorry for the wonky formatting. First time posting and I didn’t see way to preview :man_shrugging:


#3

You had most of it right :slight_smile: Just remember that it’s a single backtick for inline code blocks, and three backticks for multiline code blocks :023:

(The preview pane should be on the right while you are typing btw…)


#4

Ah, thanks @AstonJ -

For me, I only see this on the right hand side as I type

Welcome to the Elixir Forum — thanks for contributing!

Does your reply improve the conversation in some way?

Be kind to your fellow community members.

Constructive criticism is welcome, but please criticize ideas, not people.

For more, see our community guidelines. This panel will only appear for your first 2 posts.

But now I see that if I hit the x in the top right that a magical preview panel will appear. I’m guessing you haven’t seen the overlay I pasted above in quite some time :laughing:


#5

Yep, you have to close that notice to see the preview pane :003:


#6

Distillery custom commands run in the scope of distillery, not your application (I.E. this is before your application or its configuration environment is ever loaded), so you need to encode the information in that a different way, however I think that is still the wrong place to do anything that touches the Repo, I personally prefer to do things like run migrations upon every application load and load things like seeds via an explicit call during some setup phase.


#7

Thanks for the reply @OvermindDL1, I see what you are saying. I think that outlines essentially what I’m trying to do here. Ideally, I’d have the migrations run whenever the app boots and the seeds would be done once per environment during its setup. Or in our case, just reset the db and run both migrations and seeds any time seed data changes (which still shouldn’t be too often), since it’s all read-only data. This is what we’re doing actually, but we’re falling back to generating sql files using mix dump and pg_dump on a dev machine. Reading the Distillery documentation makes it seem like we can eliminate those steps by creating a custom command that executes some functions in an elixir module that actually runs the migrations and seeds themselves instead of dumped versions.

It seems that the problem is in making the repo configuration available to the ReleaseTasks module when I try to run the command from the terminal. I’ve tried following Distillery documentation about 8 different ways pre 2.0 and about 8 different ways post 2.0 but am clearly still not doing something right. It might not even have anything to do with being on Windows, rather simply my misuse of the command or an incorrect assumption I’m making about making that configuration available. But I’m all out of ideas and leads at this point :confused: Hoping someone can follow the steps above to reproduce the issue and be like, “Oh, he’s missing this thing here” or “Hmm, he’s doing everything right but it seems like an issue specific to doing this on Windows”, or something. I understand it’s probably an annoying ask for many who may have an answer as it seems like most everyone is on unix-based OSes, but I gotta try :smiley:


#8

Hey there.

Maybe I am your hero.
In the past we encountered all the problems u did.
Your post inspired me to give it another try (Windows - Distillery - Migrations / Custom Commands / Hooks)

And …
Finally …
It works!

System configuration:

Windows 10 Pro x64
Elixir 1.7.4
Erlang/OTP 21 (compiled w/ 19)
Phoenix 1.4
Ecto 3.0.3
Distillery 2.0.12
PostgreSQL 10

Problem Solving Workflow:

  1. In the "ReleaseTasks-Module" you need to change ":ecto" to ":ecto_sql" in your constant "@start_apps"
  2. In generel, configuration in config-Files doesnt work for me! (Problem: the program cant find the DB-Connection-Fields like ":database" for example! But! When you use init callback of repo it works (and… Of course it also with with dynamic loading -> System.get(…))
  3. In the "ReleaseTasks-Module: The repo Migrator needs more then "pool_size: 1" to perform tasks parallel (at least 2). So set it to 2: "repo.start_link(pool_size: 2)"

For easy comparison i created a working repository:

Have a good day!

UPDATE:
though, when i go for ecto Version “2.X” i run into same problem you mentioned…
its a big deal to go for Version 3, for example when you have to use “mssql_ecto”, which needs ecto Version “2.X” …


#9

Thank you for giving this a go Ibormann! I’ll definitely be looking into the repo you set up, and maybe this warrants a push to upgrade to Ecto 3.x. I’ll check back in later to report how it goes!


#10

its seems we are pretty alone with this situation. I guess not many developers need to use Distillery/ Elixir in a windows based environment. Until now, I invested a few more hours to get it working with ecto 2. No way.
Same story: the release can’t find the repo configuration…
Maybe there is someone who can explain where is the difference between working example with ecto 3 and not working ecto 2.


#11

Try adding Application.load(:my_app) at the beginning of start_services() in release_tasks.ex