sezaru

sezaru

How to generate index migration queries from resource at runtime?

I have a system that needs to bulk load a bunch of data from time to time.

One way to speed that load process is to delete some of the indexes for that table an then, after the load is done, re-create it.

I can do that manually via psql, but an Ash Resource already has all the information about the index, so I was wondering if there is some way for me to generate, at runtime, the same queries ash_postgres.generate_migrations generates for the resource indexes.

In other words, I want to generate create index ... and drop index ... queries for each index and identity I have in my resource so I can execute it at runtime using Repo.query

Most Liked

zachdaniel

zachdaniel

Creator of Ash

:thinking: I have no idea if this is really what you’ll end up wanting, but we do have some tooling around this kind of thing. What you can do is this:

snapshot =
  YourApi
  |> AshPostgres.MigrationGenerator.take_snapshots(YourRepo, [TheResource])
  |> Enum.at(0)

snapshot_without_indices =
  some_custom_code_to_remove_indices

operations = AshPostgres.MigrationGenerator.get_operations_from_snapshots([snapshot], [snapshot_without_indices])

{up, down} = AshPostgres.MigrationGenerator.build_up_and_down(operations)

# do something with up/down here

Thinking about this, I don’t see how you could really do what you want that way…we generate ecto migration code. You’d have to define an ecto migration module and run it. If thats on the table then you could use up to remove everything and down to add them back :person_shrugging:

sezaru

sezaru

Hey @zachdaniel , here is a quick update regarding this issue:

I created a small resource to make troubleshot easier:

defmodule Pacman.Markets.Entity2 do
  @moduledoc false

  use Ash.Resource,
    data_layer: AshPostgres.DataLayer

  attributes do
    uuid_primary_key :id
  end

  postgres do
    table "entities"

    repo Pacman.Repo
  end

  identities do
    identity :unique_id, [:id]
  end
end

When I run the take_snapshot code, I get:

iex(87)> snapshot = Pacman.Markets |> AshPostgres.MigrationGenerator.take_snapshots(Pacman.Repo, [Pacman.Markets.Entity2]) |> Enum.at(0)
%{
  attributes: [
    %{
      default: "fragment(\"uuid_generate_v4()\")",
      size: nil,
      type: :uuid,
      source: :id,
      references: nil,
      primary_key?: true,
      allow_nil?: false,
      generated?: false
    }
  ],
  table: "entities",
  hash: "8123997EA2DF402FDFB3E8145AF8666DCD0894EEF02C235DE06B42D47AF3457A",
  repo: Pacman.Repo,
  schema: nil,
  identities: [
    %{
      name: :unique_id,
      keys: [:id],
      index_name: "entities_unique_id_index",
      all_tenants?: false,
      base_filter: nil
    }
  ],
  base_filter: nil,
  multitenancy: %{global: nil, attribute: nil, strategy: nil},
  check_constraints: [],
  custom_indexes: [],
  custom_statements: [],
  has_create_action: false
}

If I just run the rest of the code, I will get the error shown above, to make it work I needed to do the following changes:

  1. Clean the attributes field array since I don’t need it and it will break;
  2. Replace the repo field value from atom to string;
  3. In the identities field, I changed the name field value from atom to string;
  4. In the identities field, I changed the keys field values from atom to string;

After these changes, my snapshot changed to the following one:

%{
  attributes: [],
  table: "entities",
  hash: "8123997EA2DF402FDFB3E8145AF8666DCD0894EEF02C235DE06B42D47AF3457A",
  repo: "Pacman.Repo", # CHANGED
  schema: nil,
  identities: [
    %{
      name: "unique_id", # CHANGED
      keys: ["id"], # CHANGED
      index_name: "entities_unique_id_index",
      all_tenants?: false,
      base_filter: nil
    }
  ],
  base_filter: nil,
  multitenancy: %{global: nil, attribute: nil, strategy: nil},
  check_constraints: [],
  custom_indexes: [],
  custom_statements: [],
  has_create_action: false
}

After these changes, the rest of the code works great:

iex(90)> snapshot_without_indices = %{snapshot | custom_indexes: [], identities: []}
%{
  attributes: [],
  table: "entities",
  hash: "8123997EA2DF402FDFB3E8145AF8666DCD0894EEF02C235DE06B42D47AF3457A",
  repo: "Pacman.Repo",
  schema: nil,
  identities: [],
  base_filter: nil,
  multitenancy: %{global: nil, attribute: nil, strategy: nil},
  check_constraints: [],
  custom_indexes: [],
  custom_statements: [],
  has_create_action: false
}
iex(91)> 
nil
iex(92)> operations = AshPostgres.MigrationGenerator.get_operations_from_snapshots([snapshot], [snapshot_without_indices])
[
  %AshPostgres.MigrationGenerator.Operation.RemoveUniqueIndex{
    identity: %{
      name: :unique_id,
      keys: [:id],
      index_name: "entities_unique_id_index",
      all_tenants?: false,
      base_filter: nil
    },
    schema: nil,
    table: "entities",
    multitenancy: %{global: nil, attribute: nil, strategy: nil},
    old_multitenancy: %{global: nil, attribute: nil, strategy: nil},
    no_phase: true
  }
]
iex(93)> 
nil
iex(94)> {down, up} = AshPostgres.MigrationGenerator.build_up_and_down(operations)
{"drop_if_exists unique_index(:entities, [:id], name: \"entities_unique_id_index\")\n",
 "create unique_index(:entities, [:id], name: \"entities_unique_id_index\")\n"}

So, the problem is that `AshPostgres.MigrationGenerator expects that some values are string and try to convert it to an atom, but in this case they are already an atom.

An easy solution for that is just to add the following function to AshPostgres.MigrationGenerator:

  defp maybe_to_atom(value) when is_atom(value), do: value
  defp maybe_to_atom(value), do: String.to_atom(value)

And replace all String.to_atom calls in that module to use that function.

After doing that the original snapshot starts working great.

Would you accept an PR with that change?

sezaru

sezaru

Thanks @zachdaniel , now the migration is perfert!

Also, just in case someone wants to do something similar to this, this is the code I’m using to actually run the up and down variables generated from the code:

  import Ecto.Migration

  args = {self(), Repo, Repo.config(), nil, :forward, :down, %{level: :info, sql: true}}

  {:ok, runner} =
    DynamicSupervisor.start_child(Ecto.MigratorSupervisor, {Ecto.Migration.Runner, args})

  Ecto.Migration.Runner.metadata(runner, [])

  # NOTE: This will run the down migration
  Code.eval_string(down, [], __ENV__)
  Ecto.Migration.Runner.flush()

  # NOTE: This will run the up migration
  Code.eval_string(up, [], __ENV__)
  Ecto.Migration.Runner.flush()

  Agent.stop(runner)

Where Next?

Popular in Questions Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
nobody
How to bind a phoenix app to a specific ip address? could not find anything about that, nowhere, unfortunately, but for me this is quite...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> somethi...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
komlanvi
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
New
yawaramin
In the Dialyzer docs ( dialyzer — OTP 29.0.2 (dialyzer 6.0.1) ), there is a way to turn off a specific warning for a function: -dialyzer...
New
jaysoifer
Is there a way to rollback a specific migration and only that one (“skipping” all the other ones)? Would mix ecto.rollback -v 200809061...
New

Other popular topics Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29603 241
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement