Statuses of multiple "N_statuses" tables have to be hardcodes throught a project. How to improve this?

I have multiple, around 10 or more, tables with statuses such as “user_statuses”, “article_statuses” and so on. In the code, in Phoenix project, I have to refer to statuses as hard-coded strings:

res = where(Article, [it], it.status_name == "active")
|> # ............

I want to improve that somehow and in a simple way. An improvement that in any way that would reduce the possiblity of making a mistake in spelling a status.

I figure I’d have to mirror of the statuses in each model, wrap each one, as functions. It’ll be tedious, but no matter how you slice it, they’ll have to be hard-coded somewhere.

def status_active, do: "active"

And thus:

res = where(Article, [it], it.status_name == Article.status_active())
|> # ............

Your ideas?

I don’t consider using third-party libraries for that.

In similar circumstances I typically create one function with multiple heads like:

@valid_status [“active”, “passive”]
def status(status_string) when status_string in @valid_status, do: status_string
def status(status_atom) when is_atom(status_atom), do: status_atom |> Atom.to_string |> status
def status(other), do: {:error, other}

This has the limitation that the string has to exactly match but if its not user input that is often enough.

You can also do some mapping if it fits better:

@valid_status %{
  active: “active”,
  passive: “passive”
}

# Return the value or raise if the value isn’t valid
def status(status) do
  Map.fetch!(@valid_status, status)
end
1 Like

If I update status or statuses in a database, I’ll have to remember to update them in the code accordingly. Can a test, unit or of other type, for that be created? How? And it should be without mocking, to avoid copy-paste of the statuses yet into the tests.

Since the codes are already in the database then some other options are:

  1. Populate an ETS table with the codes from the database when the Repo starts
  2. Populate :persistent_term if you’re on a recent OTP release. This works well unless you frequently update the codes

Both of these approaches would require synchronisation if you change the code table in the running app. I used both approaches and depending on the frequency of change it works well.

If the rate of change of the code table is linked to the frequency of releases of your app, you can also get the code table from the database at compile time and bake them into the code per the previous message. This assumes you sync the code table in production by some external means.

Rereading your original post and noting you’re doing the check in a query i think I would use the database to do the check:

  1. Use an enum type to maintain the status codes or
  2. Join to a lookup table or
  3. Add a function like is_active() to the database and use that (probably what I would use)

“N_statuses” tables == lookup tables. That’d a question of how to store that in a db.

No, I’d be over-complicated. As I’ve pointed out – in a simple manner.

My question now is this also:

If I update status or statuses in a database, I’ll have to remember to update them in the code accordingly. Can a test, unit or of other type, for that be created? How? And it should be without mocking, to avoid copy-paste of the statuses yet into the tests.

IMO if you’re regularly referencing things from those tables in code - for instance, the “active” filter in your code sample - they don’t belong in a database table at all.

If a new row was added to one of the status tables, would it be useful without code changes?

If you’re using Postgresql, consider using the database’s built-in enumerated types. Also consider the newly-added support for parametrized types in the latest Ecto.

And if I’m not using Postgresql?
Moreover, my question isn’t about a way of how to store data on a database level properly. “statuses” are stored in the lookup tables in my case, that’s it.

@romik if I understood your problem correctly, one thing you can use is to define Ecto.Types, instead of strings, and map the type to atoms. Underneath they’ll still be strings in the db but they’ll allow you to automatically cast between them, when retrieving them from the database and when querying. The Ecto.Type will have some similarities with what @kip mentioned but it will be a proper Ecto.Type so more consolidated.

It will then raise if you use a non valid atom/string in the query.
You would probably define one ecto type for each table statuses unless they are the same. So you would have a type User.Statuses.Type, Article.Statuses.Type, etc.

An example would be:

defmodule User.Activity.Type do
  @behaviour Ecto.Type

  @type t() :: :open_duel | :tournament | :pod_queue | :starting_pod | :pod | :enqueued | :duel | :starting_duel
  
  def type, do: :string

  @valid_types [:open_duel, :tournament, :pod_queue, :starting_pod, :pod, :enqueued, :duel, :starting_duel]
  @valid_string Enum.reduce(@valid_types, [], fn(t, acc) -> [Atom.to_string(t) | acc] end)
  @valid_map Enum.reduce(@valid_types, %{}, fn(t, acc) -> Map.put(acc, Atom.to_string(t), t) end)

  def load(data) when is_binary(data) and data in @valid_string, do: {:ok, @valid_map[data]}
  def load(data) when is_atom(data) and data in @valid_types, do: {:ok, data}
  def load(_), do: :error

  def cast(data) when is_atom(data) and data in @valid_types, do: {:ok, data}
  def cast(data) when is_binary(data) and data in @valid_string, do: {:ok, String.to_atom(data)}
  def cast(_), do: :error

  def dump(data) when is_atom(data) and data in @valid_types, do: {:ok, Atom.to_string(data)}
  def dump(data) when is_binary(data) and data in @valid_string, do: {:ok, data}

  def embed_as(_), do: :dump
  def equal?(term_1, term_2) when is_binary(term_1) and is_atom(term_2) do
    Atom.to_string(term_2) === term_1
  end

  def equal?(term_1, term_2) when is_binary(term_2) and is_atom(term_1) do
    Atom.to_string(term_1) === term_2
  end

  def equal?(term_1, term_2), do: term_1 === term_2
end

But changed to your needs.
This allows you to cast/validate types before hand, like by calling User.Activity.Type.cast("some_value") either as an atom or string. It will return :error if it’s not a valid type.

And it allows you to use either the atom or string form in queries, so for instance

User.Activity |> where([a], a.type == ^:open_game) |> Db.Repo.all
(you need to pin atoms when writing them literally in the query, but if it’s in a variable you’ll pin the variable anyway).

If the value isn’t acceptable in the query it will raise, so you’ll get an error such as:

** (Ecto.Query.CastError) iex:10: value "PLAYGROUND"inwhere cannot be cast to type User.Activity.Type in query:

You can get fancier with using some more meta programming and generating function names that give you the correct type, like you have with Article.status_active() but I’m not sure there’s anything to gain there, unless you plan/or want to be able to change the status that corresponds to status_active.

Like using the previous type and adding something such as:

Enum.each(@valid_types, fn(value) ->
    name = String.to_atom("status_#{value}")
    def unquote(name)(), do: unquote(value)
end)

In that module that would translated to functions name status_open_duel, etc. So you could use User.Activity.Type.status_open_duel to get back :open_duel. To make it more useful you would probably want to change the @valid_types to be instead

@valid_types [{:open_duel, :open_duel}, {:tournament, :tournament}, {:pod_queue, :pod_queue}, {:starting_pod, :starting_pod}, {:pod, :pod}, {:enqueued, :enqueued}, {:duel, :duel}, {:starting_duel, :starting_duel}]
@valid_string Enum.reduce(@valid_types, [], fn({_, t}, acc) -> [Atom.to_string(t) | acc] end)
@valid_map Enum.reduce(@valid_types, %{}, fn({_, t}, acc) -> Map.put(acc, Atom.to_string(t), t) end)

Enum.each(@valid_types, fn({description, value}) ->
    name = String.to_atom("status_#{description}")
    def unquote(name)(), do: unquote(value)
end)

This would then allow you to change :open_duel, to be instead :waiting_duel, while keeping User.Activity.Type.status_open_duel() throughout the codebase. But I kinda prefer to forget that and just use directly the status.

thanks, let me see.

  • And I update a status in a database, and forget to update it in the code, it’d all break down.

  • How is better as simply using statuses as pure strings, without any of that code? In both cases an exception would be raised in runtime. Then, what’s the difference?

The solution is to store these at the database level and use the database to ensure integrity, but by the previous posts I got the idea you didn’t want that but instead you wanted it at the application level. If you’re going to be fiddling around the DB without going through the application then I’m not sure how you expect it to be aware of changes you could possibly make there?

How would you solve it in another language/framework given your constraints?

  • And I update a status in a database, and forget to update it in the code, it’d all break down.

No, you would get an exception when retrieving any record with an invalid code, because Ecto wouldn’t be able to load the value into an acceptable term per your Ecto.Type.

  • How is better as simply using statuses as pure strings, without any of that code? In both cases an exception would be raised in runtime. Then, what’s the difference?

Without an Ecto.Type to properly cast the values the string will not raise any error since the underlying value in the DB is a string, the query would work just normally, but would not raise an exception on “making a mistake in spelling a status” which was your originally ask, it would just not return results.

If you have a test suite that simply executes these queries, and you misspell a status on your code base, then you’ll see these errors during test execution (you don’t need to test the queries, simply any path that would invoke a query with an invalid status value) - and if you don’t have test suites then you would see errors in production instead of the query silently returning zero records (for a misspelled status that doesn’t exist on the DB), and if you don’t have tests you’ll probably at least run the query once manually, and you would see the error once again.

I want:

  • statuses in both db and app

  • they always, or at application startup will do also, to be syncronized; statuses in a db are the main ones

  • if I spell a staus incorrectly in Elixir code, it’ll throw an exception during compilation, or when running tests. That is, it’ll signal me that there’s an inconsistency between the statuses in a db and code.

  • solution simple enough; no third-party libraries

And

  • statuses should only be present once in the code or yml file, in either; in the tests - not; the tests should somehow try to use the statuses from the dev/prod environment. But how, is it possible at all?

Why the no, if you’re re-confirming what I’ve said? Namely, an exception will be thrown at runtime, or a nill will be returned at runtime.

That’s not what I need - I don’t need at runtime. For I need it to be raised during compilation, or during running the tests, at least, so that I’ll know that the statuses in the code have not been syncronized with the ones in a db.

Because an exception is traceable while nil is a valid return value for a query returning a single row.

I don’t have a divining ball so I can’t know what you need or not when you keep changing the description of what you want to achieve. Still you haven’t explained how you would go about doing it in other languages.

Again, if you want status both in DB and App first how will you import them to the DB, I imagine through a migration, if you have the data specified in an YML file - you can both read it during compilation and during the initial migration - but then the question becomes, what happens once you need to change it. Anyway.

I think that with Elixir you’re in a better position to achieve what you want, but it won’t be simple as you wish, if you know some other language or ecosystem you can show with an example so we can see how to achieve the same here.

You have metaprogramming, so you can define things at compile time, I gave you an example of simple in module code generation. You can bring it further and read a file during compilation and build your functions/module with data contained in that file (i.e. your YML file, you still need to worry though about loading that into the db, and again it doesn’t protect you from random changes you might do directly on the DB that you mentioned so if that’s a worry…).

Other option but really I don’t think it’s feasible is to access the DB in question during compilation. You have a host of options for starting arbitrary apps (supervision trees, etc) at any point, you could retrieve the needed data from the DB during compilation and use it to create your module/functions through metaprogramming again but instead of reading the file now you involve network and production database access to your build chain.

I know there’s ways to compile things on the fly but it’s an area I didn’t try yet. This could be feasible, by adding a process to the main app supervision tree, that would query the db on startup, and dynamically compile a module. (or if in an umbrella app, have a dedicated “bootstrapper” app that does all that, including running migrations, etc, before starting the “proper” app elements)

There’s other option, where you can have the same as the previous one, but instead of dynamically compiling a module, create either a public ETS table, or use persistent term, where you query the DB right at the beginning of your supervision tree startup, populate the ETS table (or persistant terms) with a key-value scheme that allows you to later on retrieve them accordingly. So perhaps your k-vs would be {:status, :post} → [:published, :etc] and you could then from the Ecto.Type validate them dynamically.

So for compilation errors you need to do the work at compilation time, for test errors you can use these last 2 options - but your test database will have itself to be in synch with the status that will be in production otherwise there’s no guarantees.

Have fun

In what other languages? Why?

For the sake of God, answer succinctly.

:clown_face:

You should… ecto_enum is done for this problem.

If people take time to answer, but You reply like this…

… You might have this kind of short answer

Ecto 3.5 will have native support for what ecto_enum does.

1 Like