Pretty straight forward situation. I added a field in a new migration and added the field to the User module’s schema.
Now a previous migration fails to run saying:
** (Postgrex.Error) ERROR 42703 (undefined_column): column u0.is_bot does not exist
(db_connection) lib/db_connection.ex:1406: DBConnection.prepare_declare/4
(elixir) lib/stream.ex:1270: anonymous fn/5 in Stream.resource/3
(elixir) lib/stream.ex:1433: Enumerable.Stream.do_each/4
(elixir) lib/stream.ex:806: Stream.do_transform/8
(elixir) lib/stream.ex:1433: Enumerable.Stream.do_each/4
(elixir) lib/stream.ex:591: Stream.run/1
(ecto) lib/ecto/adapters/sql.ex:576: anonymous fn/3 in Ecto.Adapters.SQL.do_transaction/3
(db_connection) lib/db_connection.ex:1275: DBConnection.transaction_run/4
The migration code that can’t run looks like:
def up do
Repo.transaction(
fn ->
User
|> Repo.stream()
|> Stream.each(fn user ->
# Do something with `user`.
end)
|> Stream.run()
end,
timeout: :infinity
)
end
Any suggestions on how to fix this situation? Locally, I can comment out the is_bot field declaration and run the migrations, but this has broken CI completely for me.
What do you guys think of modifying my migration script to only select the fields it needs, thus preventing is_bot from being attempted?
Yeah I can fallback to raw sql, but I was wondering why this seemingly common case isn’t supported easily by Ecto. Most likely, I’m messing up somewhere.
A migration should be reversible, why does previous migrations see is_bot? Also why use def up and not def change? def up should have a corresponding def down.
If You could show your last migration file (the one that add is_bot attribute), that would help.
in that old migration you are using the User schema, that now has changed… it’s a big no no to not have isolated migrations - as you are experiencing
either rewrite the migration to not access the User schema, or make a copy of that schema at the time of migration, rename it, and include it in the migration…
I think @outlog meant something like having a UserBeforeChange schema, which does not include is_bot attribute, and use it instead of User in your migration.
The other way would be to replace
User
|> select([:id, :email, :onboarded, :roles])
|> Repo.stream()
...
What’s wrong with defining the User schema that you want within the migration file? This would allow you to be insulated from future changes to the User schema and makes the migrations stand-alone.
defmodule MyApp.Repo.Migrations.ChangeUsers
defmodule User do
use MyAppWeb, :schema
schema "users" do
field :name
field :email
end
end
def up do
Repo.transaction(
fn ->
User
|> Repo.stream()
|> Stream.each(fn user ->
# Do something with `user`.
end)
|> Stream.run()
end,
timeout: :infinity
)
end
end
I believe that something along the lines is the general recommended approach, although I’m having trouble finding a mention in the docs. Maybe it was in the ecto e-book.
It’s a nuisance. Every time we write a migration we’ll have to explicitly create a schema with the fields or else we might regress and have to add it when a bug happens like today.
I still don’t understand why in my previous post .update_all is calling the schema in my user module.
I suspect (with limited confidence) that in the code above, user is of type %User{}. ie, your update_all is updating only one row and its using the schema from %User{}
