Introducing Enumex: an advanced enum library

Eiji · November 2, 2019, 2:27pm

Yes, one more library for working with enums and also my 2nd enum library!

Introduction

My first library was really simple and was mainly intended for personal usage. I did not even expected that other people would use it. This one is completely different as I found that there is still something to do in this place. First of all this library was intended to be ecto-postgres-enum ~> 2.0, but way too many things changed.

The first thing to note is that it’s not intended to work only for PostgreSQL database and it even does not require any database. There are lots of helpful features which could be used as same as ecto without ecto_sql and that’s mainly why it’s not just another version of ecto-postgres-enum.

Try it

For now this library depends on master branch of ecto_sql, so I’m unable to release it as a hex package. For now please give it a try by cloning a GitLab repository:

This is only a release candidate and gives only preview on what I would like to provide in final release. Based on contribution it may change at any time before 1.0.0 stable release. For each release candidate I would add a git tag.

Migrations

However the true power of this library comes with ecto_sql support. The one of main features are migrations … fully reversible migrations! It’s not only about create and drop enum, but also add and drop value. I have written test to ensure that every migration does not cause problem with existing databases, but saying that is like don’t saying anything … You may wonder what happens if we would like to drop value which is already used, but this use case is also covered and tested!

My code for migrations is based on few articles which describes a safely work on database enum and here is one of them:

How does it work in migration file? It’s as simple as calling just one function!

Enumex.Ecto.Migration.drop_value(adapter, prefix, name, repo, value, default)

At this point you may be confused … How could it be reversible? Of course it’s not possible to turn back value for each row, but this is normal and same result you would have working in raw PostgreSQL console. This library focused on what database is able to do and offer such features in as simple as possible solutions.

This one feature took for me lots of time especially that ecto_sql previously blocked me in this specific scenario. Fortunately thanks to @josevalim this has been solved in master branch of ecto_sql.

Tasks

Of course there is more interesting feature … What would you say if you could simply diff updated your enum in Elixir just now with values which database is storing? Yes, it’s possible! Take a look at enumex.gen.migration mix task. It’s not only automatically detecting value add and drop, but also rename!

Other features

I could describe here all of features, but it could turn into a series of blog posts, so I simply list them all:

Constants-like macros
In response to: Using Enums as constants · Issue #87 · gjaldon/ecto_enum · GitHub
Built-in absinthe support! Everything you need is to import type from YourEnum.Absinthe module.
In response to: Dynamically generate graphql types · Issue #78 · gjaldon/ecto_enum · GitHub
Migrations - except those already described there are few more helpful functions.
Ecto support
Lots of other debug functions, guards, work with indexes and more …

Database support

For now I’m only supporting PostgreSQL, but I have a plans to change it in next release candidates. However my knowledge about other databases is limited and it could take for me some time. I have take a look at MySQL and … do I need to comment it?

This library is intended for database like PostgreSQL which provides a reasonable API. Ok, but what if other databases? I have an idea here. I can implement a Enumex.Adapters.SQLTable which would provide a __using__/1 macro for final adapters. In this way I would be able to add support not only for MySQL, but SQLite also does not looks so problematic with such solution.

Anyway as said I have not much experience with other databases (at least in enum topic), so I it would be really nice if someone could help with adding support for other adapters.

Future plans

As already said I’m planning to add at least one helper module called Enumex.Adapters.SQLTable and I would write (just for an example) an implementation for PostgreSQL so other developer may see how to deal with it. This mean that there would be at least one release candidate. What will come later I’m not sure, but I would really like to release it on hex as soon as possible even if it would be only for release candidate test purposes.

Have a nice day!

Hi, originally I wanted to place it in mailing lists as proposition, but found t…hat's more like bug, because it does not work as same as before `Elixir` release `1.9.0`. First of all this does not produces any compile-time warning: https://github.com/elixir-ecto/ecto/blob/c0d39069cdc92dcf34f1a528bd8eaf52f328d263/lib/mix/tasks/ecto.gen.repo.ex#L85-L93 I believe that typical beginner may not even correct this. Just btw. I would like to ask if we should add warning in `Mix.Config.__using__/1` macro. The bigger problem is here: https://github.com/elixir-ecto/ecto/blob/c0d39069cdc92dcf34f1a528bd8eaf52f328d263/lib/mix/tasks/ecto.gen.repo.ex#L54-L55 If we would have `import Config` then repo configuration would just not be added - this part looks like a bug for me as general behavior (adding config) is changed (not working as expected). I suggest to add check like: ```elixir Version.match? "1.9.0", Mix.Project.config()[:elixir] ``` and store first config line in a variable. Is it ok to make PR for this one right now or it should be discussed on mailing lists before it?

The blocking flush() bug and new execute usage:

github.com/elixir-ecto/ecto_sql

Migration change, ecto.rollback and flush problem

opened 04:03AM - 28 Oct 19 UTC

closed 10:09AM - 01 Nov 19 UTC

Eiji7

This situation happen to me when working with reversible migrations, so instead …of `down`, `execute/1` and `up` I was simply using `change` and `execute/2`. I noticed that one part requires first to be finished, so I was in need to add `flush()` call. With `ecto.migrate` there was no problem. In `ecto.rollback` commands were althrough in reverted order, but not as I imagined … When I had `first part` of migration, `flush()` call and `second part` of migration then after using `ecto.revert` I got reversed `first part` of migration, `flush()` call and reversed `second part` of migration. Should it not be like: reversed `second part` of migration, `flush()` call and reversed `first part` of migration instead? That's how I understood it after reading documentation. In order words `flush()` in `ecto.rollback` task works like `flush()` in `ecto.migrate` task, but each part (separated by `flush` call) have reversed commands and the parts are executed with not reversed order (like in `ecto.migrate`). This is really easily reproducable by such simple migration: ```elixir defmodule Example.Repo.Migrations.Test do use Ecto.Migration @migrate_first "select 'first part of ecto.migrate';" @migrate_second "select 'second part of ecto.migrate';" @rollback_first "select 'first part of rollback.migrate';" @rollback_second "select 'second part of rollback.migrate';" def change do execute(@migrate_first, @rollback_second) flush() execute(@migrate_second, @rollback_first) end end ``` Output with `flush()` call commented: ```elixir $ mix ecto.migrate 05:14:25.060 [info] == Running 20191028033147 Example.Repo.Migrations.Test.change/0 forward 05:14:25.063 [info] execute "select 'first part of ecto.migrate';" 05:14:25.064 [info] execute "select 'second part of ecto.migrate';" 05:14:25.067 [info] == Migrated 20191028033147 in 0.0s $ mix ecto.rollback 05:14:25.166 [info] == Running 20191028033147 Example.Repo.Migrations.Test.change/0 backward 05:14:25.166 [info] execute "select 'first part of rollback.migrate';" 05:14:25.167 [info] execute "select 'second part of rollback.migrate';" 05:14:25.168 [info] == Migrated 20191028033147 in 0.0s ``` Output with `flush()` uncommented: ```elixir $ mix ecto.migrate 05:13:22.580 [info] == Running 20191028033147 Example.Repo.Migrations.Test.change/0 forward 05:13:22.583 [info] execute "select 'first part of ecto.migrate';" 05:13:22.584 [info] execute "select 'second part of ecto.migrate';" 05:13:22.587 [info] == Migrated 20191028033147 in 0.0s $ mix ecto.rollback 05:13:22.682 [info] == Running 20191028033147 Example.Repo.Migrations.Test.change/0 backward 05:13:22.682 [info] execute "select 'second part of rollback.migrate';" 05:13:22.683 [info] execute "select 'first part of rollback.migrate';" 05:13:22.684 [info] == Migrated 20191028033147 in 0.0s ``` I may be wrong, but for me it looks like a bug. What do you think about it?

github.com/elixir-ecto/ecto_sql

First version of dynamic command for Ecto.Migration

elixir-ecto:master ← Eiji7:master

opened 05:48AM - 29 Oct 19 UTC

Eiji7

+83 -7

- [x] Added `Ecto.Migration.dynamic/1` macro. - [x] Added integration test for …ensuring correct order. - [x] Added optional support for passing an anonymous function with one arity. - [x] Added special function clausule in `Migration.Runner` in order to **not afftect** currect adapters implementations. - [x] Documentation and specs - [x] Ensure that `{:dynamic, func}` command is self-reversible in runner. **TODO**: - [x] Discuss the implementation details Related issue: #157

A confusing error of wrongly emitted Erlang AST:

github.com/elixir-ecto/ecto_sql

Got Postgrex.Error prepared statement "ecto_insert_posts" does not exist in ecto_sql tests

opened 06:18PM - 31 Oct 19 UTC

closed 06:54PM - 03 Jun 20 UTC

Eiji7

### Environment I have up to date `openSUSE Tumbleweed`. However I have insta…lled `Erlang`, `Elixir` and `PostgreSQL` using [asdf](https://github.com/asdf-vm/asdf) (extendable version manager). ```bash $ asdf current elixir 1.9.2-otp-22 (set by …) erlang 22.1.5 (set by …) postgres 12.0 (set by …) $ elixir -v Erlang/OTP 22 [erts-10.5.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] Elixir 1.9.2 (compiled with Erlang/OTP 22) ``` Steps to reproduce: 1. Install `asdf` with 3 plugins: `erlang`, `elixir` and `postgres`. 2. Install latest versions (as above) from all of those plugins 3. Ensure you have no other instances of `PostgreSQL` running 4. Run `pg_ctl start` 5. Create a `UTF-8` database replacing `username` with name of your `Linux` account (`asdf` prepares account for easy `dev` work) 6. Clone `ecto_sql` 7. Run few times tests Here is an `SQL` I have used in order to create database: ```sql create database "username" with owner "username" encoding 'UTF8' LC_COLLATE = 'pl_PL.UTF-8' LC_CTYPE = 'pl_PL.UTF-8' TEMPLATE template0; ``` > Based on: https://support.helpspot.com/index.php?pg=kb.page&id=467 Here is my test command: ```bash PG_URL="username:@127.0.0.1" ECTO_ADAPTER=pg mix test ``` All of dependencies are up to date as today I was looking for this error on my fork of `master` branch. Here is a complete list of all dependencies: ```bash * connection 1.0.4 (Hex package) (mix) locked at 1.0.4 (connection) a1cae722 ok * telemetry 0.4.0 (Hex package) (rebar3) locked at 0.4.0 (telemetry) 8339bee3 ok * decimal 1.8.0 (Hex package) (mix) locked at 1.8.0 (decimal) ca462e0d ok * db_connection 2.1.1 (Hex package) (mix) locked at 2.1.1 (db_connection) a51e8a2e ok * ecto 3.3.0-dev (https://github.com/elixir-ecto/ecto.git) (mix) locked at 74cd9f5 ok * postgrex 0.15.0 (Hex package) (mix) locked at 0.15.0 (postgrex) dd534916 ok * myxql 0.2.4 (Hex package) (mix) locked at 0.2.4 (myxql) 010c3a14 ok ``` ### Current behavior Here is the error which happen locally and on CI: ```elixir 1) test insert autogenerates for custom type (Ecto.Integration.RepoTest) deps/ecto/integration_test/cases/repo.exs:226 ** (Postgrex.Error) ERROR 26000 (invalid_sql_statement_name) prepared statement "ecto_insert_posts" does not exist code: post = TestRepo.insert!(%Post{uuid: nil}) stacktrace: (ecto_sql) lib/ecto/adapters/sql.ex:631: Ecto.Adapters.SQL.raise_sql_call_error/1 (ecto) lib/ecto/repo/schema.ex:651: Ecto.Repo.Schema.apply/4 (ecto) lib/ecto/repo/schema.ex:262: anonymous fn/15 in Ecto.Repo.Schema.do_insert/4 (ecto) lib/ecto/repo/schema.ex:164: Ecto.Repo.Schema.insert!/4 deps/ecto/integration_test/cases/repo.exs:227: (test) ``` possibly related: ```elixir 1) test insert and update with changeset (Ecto.Integration.RepoTest) deps/ecto/integration_test/cases/repo.exs:166 ** (Postgrex.Error) ERROR 26000 (invalid_sql_statement_name) prepared statement "ecto_update_posts" does not exist code: assert %Post{text: "y", title: "world", temp: "unknown"} = TestRepo.update!(changeset) stacktrace: (ecto_sql) lib/ecto/adapters/sql.ex:629: Ecto.Adapters.SQL.raise_sql_call_error/1 (ecto) lib/ecto/repo/schema.ex:651: Ecto.Repo.Schema.apply/4 (ecto) lib/ecto/repo/schema.ex:349: anonymous fn/15 in Ecto.Repo.Schema.do_update/4 (ecto) lib/ecto/repo/schema.ex:177: Ecto.Repo.Schema.update!/4 deps/ecto/integration_test/cases/repo.exs:179: (test) ``` As I'm able to reproduce I have inspected the error struct: ```elixir %Postgrex.Error{ connection_id: 10209, message: nil, postgres: %{ code: :invalid_sql_statement_name, file: "prepare.c", line: "502", message: "prepared statement \"ecto_insert_posts\" does not exist", pg_code: "26000", routine: "FetchPreparedStatement", severity: "ERROR", unknown: "ERROR" }, query: nil } ``` I'm not sure if that would help, but some tests are failing if developer would not create a database, but even that thing is not stopping this error from raising, so maybe it could somehow help by elimination. ### Expected behavior All tests should pass. Hope that it helps. I will still look for possible solution, but it may take some time for me as I have not yet contributing to this part of code in this repository. If you are still unable to reproduce then maybe I could help and send some inspected results that you may find interesting.

Once again I would like to thank @josevalim for all his help as without him really important parts of this library would not work. Also without it this library would would not be released so fast and most probably it could have a different form.

Feel free in case of any question. I would try to reply to all of them as soon as possible. Hope my work would help in future also in production environments.

unsek · July 6, 2020, 12:46pm

Is this still being maintained?

Eiji · July 6, 2020, 1:31pm

Yes, I have even plans for next release candidate (which most probably would be final version) when 2 things would happen:

ecto would release a new version with this PR merged
https://github.com/elixir-ecto/ecto/pull/3339
Temperature here would be lower

Lots of things happen in last 7 months and I was not focused on next release candidate.I have started working on it and already made 1/3 of things on my TODO list locally.

I can say already that final version would have 4 scenarios:

Advanced (based on Enumex.Advanced.Adapter behaviour) - that is for optimizations on db side - like support for PostgreSQL custom type (i.e. create type … syntax)
Custom - case when enum values are stored in database table - that’s my concept of custom enum il.e. Advanced scenario, but similarly to Simple useful when Enumex.Advanced.Adapter cannot be implemented. You may want to save more information in database or add for example constraints.
Simple - similar to Advanced, but for cases when implementing Enumex.Advanced.Adapter is not possible (MySQL) - you would be able to store plain integer (index in enum) or string (value in enum)
Extra support for Ecto.ParameterizedType and :enum (see PR) - due to how ecto would solve things it would work as same as Simple, but would have a bit different API (you would need to pass schema and field names instead of working on each enum module)

Each scenario is completely different and depends on few things:

If you have legacy codebase with any of such scenario
If you need to use specific database i.e. project requirements
If for some reason you need to use older versions of Ecto
If you want to use Ecto.ParameterizedType and :enum

Once next release candidate would be released I would like to know:

Do you have any other scenario (especially in prod)?
Would it work with ecto version 2.x? Originally there were no plans for it, but I will check how much work I would need to do in order to add backwards compatibility.
Do you have some other suggestions? Improvements? Bugs?