sezaru
How to generate index migration queries from resource at runtime?
I have a system that needs to bulk load a bunch of data from time to time.
One way to speed that load process is to delete some of the indexes for that table an then, after the load is done, re-create it.
I can do that manually via psql, but an Ash Resource already has all the information about the index, so I was wondering if there is some way for me to generate, at runtime, the same queries ash_postgres.generate_migrations generates for the resource indexes.
In other words, I want to generate create index ... and drop index ... queries for each index and identity I have in my resource so I can execute it at runtime using Repo.query
Most Liked
zachdaniel
I have no idea if this is really what you’ll end up wanting, but we do have some tooling around this kind of thing. What you can do is this:
snapshot =
YourApi
|> AshPostgres.MigrationGenerator.take_snapshots(YourRepo, [TheResource])
|> Enum.at(0)
snapshot_without_indices =
some_custom_code_to_remove_indices
operations = AshPostgres.MigrationGenerator.get_operations_from_snapshots([snapshot], [snapshot_without_indices])
{up, down} = AshPostgres.MigrationGenerator.build_up_and_down(operations)
# do something with up/down here
Thinking about this, I don’t see how you could really do what you want that way…we generate ecto migration code. You’d have to define an ecto migration module and run it. If thats on the table then you could use up to remove everything and down to add them back ![]()
sezaru
Hey @zachdaniel , here is a quick update regarding this issue:
I created a small resource to make troubleshot easier:
defmodule Pacman.Markets.Entity2 do
@moduledoc false
use Ash.Resource,
data_layer: AshPostgres.DataLayer
attributes do
uuid_primary_key :id
end
postgres do
table "entities"
repo Pacman.Repo
end
identities do
identity :unique_id, [:id]
end
end
When I run the take_snapshot code, I get:
iex(87)> snapshot = Pacman.Markets |> AshPostgres.MigrationGenerator.take_snapshots(Pacman.Repo, [Pacman.Markets.Entity2]) |> Enum.at(0)
%{
attributes: [
%{
default: "fragment(\"uuid_generate_v4()\")",
size: nil,
type: :uuid,
source: :id,
references: nil,
primary_key?: true,
allow_nil?: false,
generated?: false
}
],
table: "entities",
hash: "8123997EA2DF402FDFB3E8145AF8666DCD0894EEF02C235DE06B42D47AF3457A",
repo: Pacman.Repo,
schema: nil,
identities: [
%{
name: :unique_id,
keys: [:id],
index_name: "entities_unique_id_index",
all_tenants?: false,
base_filter: nil
}
],
base_filter: nil,
multitenancy: %{global: nil, attribute: nil, strategy: nil},
check_constraints: [],
custom_indexes: [],
custom_statements: [],
has_create_action: false
}
If I just run the rest of the code, I will get the error shown above, to make it work I needed to do the following changes:
- Clean the
attributesfield array since I don’t need it and it will break; - Replace the
repofield value from atom to string; - In the
identitiesfield, I changed thenamefield value from atom to string; - In the
identitiesfield, I changed thekeysfield values from atom to string;
After these changes, my snapshot changed to the following one:
%{
attributes: [],
table: "entities",
hash: "8123997EA2DF402FDFB3E8145AF8666DCD0894EEF02C235DE06B42D47AF3457A",
repo: "Pacman.Repo", # CHANGED
schema: nil,
identities: [
%{
name: "unique_id", # CHANGED
keys: ["id"], # CHANGED
index_name: "entities_unique_id_index",
all_tenants?: false,
base_filter: nil
}
],
base_filter: nil,
multitenancy: %{global: nil, attribute: nil, strategy: nil},
check_constraints: [],
custom_indexes: [],
custom_statements: [],
has_create_action: false
}
After these changes, the rest of the code works great:
iex(90)> snapshot_without_indices = %{snapshot | custom_indexes: [], identities: []}
%{
attributes: [],
table: "entities",
hash: "8123997EA2DF402FDFB3E8145AF8666DCD0894EEF02C235DE06B42D47AF3457A",
repo: "Pacman.Repo",
schema: nil,
identities: [],
base_filter: nil,
multitenancy: %{global: nil, attribute: nil, strategy: nil},
check_constraints: [],
custom_indexes: [],
custom_statements: [],
has_create_action: false
}
iex(91)>
nil
iex(92)> operations = AshPostgres.MigrationGenerator.get_operations_from_snapshots([snapshot], [snapshot_without_indices])
[
%AshPostgres.MigrationGenerator.Operation.RemoveUniqueIndex{
identity: %{
name: :unique_id,
keys: [:id],
index_name: "entities_unique_id_index",
all_tenants?: false,
base_filter: nil
},
schema: nil,
table: "entities",
multitenancy: %{global: nil, attribute: nil, strategy: nil},
old_multitenancy: %{global: nil, attribute: nil, strategy: nil},
no_phase: true
}
]
iex(93)>
nil
iex(94)> {down, up} = AshPostgres.MigrationGenerator.build_up_and_down(operations)
{"drop_if_exists unique_index(:entities, [:id], name: \"entities_unique_id_index\")\n",
"create unique_index(:entities, [:id], name: \"entities_unique_id_index\")\n"}
So, the problem is that `AshPostgres.MigrationGenerator expects that some values are string and try to convert it to an atom, but in this case they are already an atom.
An easy solution for that is just to add the following function to AshPostgres.MigrationGenerator:
defp maybe_to_atom(value) when is_atom(value), do: value
defp maybe_to_atom(value), do: String.to_atom(value)
And replace all String.to_atom calls in that module to use that function.
After doing that the original snapshot starts working great.
Would you accept an PR with that change?
sezaru
Thanks @zachdaniel , now the migration is perfert!
Also, just in case someone wants to do something similar to this, this is the code I’m using to actually run the up and down variables generated from the code:
import Ecto.Migration
args = {self(), Repo, Repo.config(), nil, :forward, :down, %{level: :info, sql: true}}
{:ok, runner} =
DynamicSupervisor.start_child(Ecto.MigratorSupervisor, {Ecto.Migration.Runner, args})
Ecto.Migration.Runner.metadata(runner, [])
# NOTE: This will run the down migration
Code.eval_string(down, [], __ENV__)
Ecto.Migration.Runner.flush()
# NOTE: This will run the up migration
Code.eval_string(up, [], __ENV__)
Ecto.Migration.Runner.flush()
Agent.stop(runner)
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance









