Shared module constants

Hi,

I’m porting an old PHP application over to Elixir and Phoenix, and I’m struggling to figure out the best way to transfer the constants that currently exist.

At the moment there are multiple files with various constants used throughout the app, for instance:

<?php
namespace Shared\Consts;

class System
{
    const AUDIT_ENTITY_REV_TYPE_DELETE = 301001;
    const AUDIT_ENTITY_REV_TYPE_INSERT = 301002;
    const AUDIT_ENTITY_REV_TYPE_UPDATE = 301003;
    const AUDIT_LOG_ACTION_CREATE = 301004;
    const AUDIT_LOG_ACTION_CUSTOM = 301005;
    const AUDIT_LOG_ACTION_DELETE = 301006;
    const AUDIT_LOG_ACTION_EDIT = 301007;
   
    const SOURCE_LVL1_API = 301023; 
    const SOURCE_LVL1_HTTP = 301024;
    const SOURCE_LVL1_EMAIL = 301025;
    const SOURCE_LVL1_SMS = 301026;
    const SOURCE_LVL1_UNKNOWN = 301027;

    const SOURCE_LVL2_STAFF_PANEL = 301028;
    const SOURCE_LVL2_STAFF_APP = 301029;
    const SOURCE_LVL2_RESELLER_PANEL = 301030;
    const SOURCE_LVL2_RESELLER_APP = 301031;
    const SOURCE_LVL2_AFFILIATE_PANEL = 301032;
    const SOURCE_LVL2_CUSTOMER_PANEL = 301033;
    const SOURCE_LVL2_CUSTOMER_APP = 301034;
    const SOURCE_LVL2_UNKNOWN = 301035;
}

Then in other files I can do:

<?php
use Shared\Const\System as SysConst;

if (SysConst::SOURCE_LVL1_API === $someOtherVariable)
{
    echo 'Starts at API';
}

Is there a way to do something similar in Elixir? Or is there a better way to store lots of constants that could be the same name, but relate to different areas?

4 Likes

The usual approach would be to use atoms inside the system, e.g. :source_lvl1_api and convert to/from integer encoding on the system boundary, if needed. This makes it easy for debugging and introspection since at runtime you have readable atoms, instead of opaque integer values flying around. You can generate the conversion functions easily with a sprinkle of macros:

values = [source_lvl1_api: 301023, ...]
for {key, value} <- values do
  def encode(unquote(key)),   do: unquote(value)
  def decode(unquote(value)), do: unquote(key)
end
9 Likes

You can also store it into a map that is returned from a function like this:

def Consts, do: %{
  AUDIT_ENTITY_REV_TYPE_DELETE: 301001,
  AUDIT_ENTITY_REV_TYPE_INSERT: 301002,
  ...
}

Being a constant object in the code with no variables it gets constructed fast, however usual things in Erlang/Elixir get optimized into the bytecode so no construction actually takes place and it gets shared easily, I’m not sure if maps do that however.

Overall @michalmuskala method is usually best, no construction time at all, however the lookup time can be linear in a match set like that so a map might be faster in lookup speed.

5 Likes

Thanks both :slight_smile:

So it looks like @OvermindDL1 idea of using a map would be the easiest way to get them across, but I’m intrigued by what you said @michalmuskala.

I’m not sure I fully grasp what you are suggesting, are you saying that the conversion would happen at the point that things are inserted and retrieved from the database? I’m struggling to figure out where those macros would live and how they would be used in other modules.

+1 to map solution since keyword list can have duplicate keys and with many constants it might become a problem down the way

I have a macro for reusing constants across multiple modules. The benefit being that they work in matches and in guards.

defmodule Constants do
  @moduledoc """
  An alternative to use @constant_name value approach to defined reusable 
  constants in elixir. 

  This module offers an approach to define these in a
  module that can be shared with other modules. They are implemented with 
  macros so they can be used in guards and matches

  ## Examples: 

  Create a module to define your shared constants

      defmodule MyConstants do
        use Constants

        define something,   10
        define another,     20
      end

  Use the constants

      defmodule MyModule do
        require MyConstants
        alias MyConstants, as: Const

        def myfunc(item) when item == Const.something, do: Const.something + 5
        def myfunc(item) when item == Const.another, do: Const.another
      end

  """
  
 defmacro __using__(_opts) do
    quote do
      import Constants
    end
  end

  @doc "Define a constant"
  defmacro constant(name, value) do
    quote do
      defmacro unquote(name), do: unquote(value)
    end
  end

  @doc "Define a constant. An alias for constant"
  defmacro define(name, value) do
    quote do
      constant unquote(name), unquote(value)
    end
  end
end
13 Likes

This is quite cool! Would be nice if it was possible to require and alias at the same time, but I think I might have a play with this as the general way to do constants. :slight_smile:

Well, in the “required” module you could implement a __using__/1 macro and use it from the requiring one. In that __using__/1 you then can of course require and alias at the same time.

1 Like

Using such macros has still the same issue - you have opaque integers passing around at runtime - this makes it really hard to debug issues, especially if dealing with large numbers.

As to when do conversions - I spoke about boundaries of the system. By that I mean every time we receive data from outside or send it out - this means params from requests, database responses, etc. For example, when using ecto, you can provide a custom type to do the conversions for you (assuming encode and decode functions as before):

defmodule Constant do
  @behaviour Ecto.Type

  def type, do: :integer

  def cast(int) when is_integer(int), do: {:ok, decode(value)}
  def cast(atom) when is_atom(atom), do: {:ok, atom)
  def cast(_), do: :error

  def load(int) when is_integer(int), do: {:ok, decode(value)}
  def load(_), do: :error

  def dump(atom) when is_atom(atom), do: {:ok, encode(value)}
  def dump(_), do: :error
end
5 Likes

I would definitely go for the approach from @michalmuskala. One problem with maps is that it only supports one way conversion (converting a const into an integer). If you need to convert an integer into a const, you’ll have to build a reversal map. This can be easily done, even during compilation time, but then the code becomes as complex if not more than Michal’s version.

Moreover, I’m not completely sure whether this map is stored in the so called “constant pool”, and if it’s not, then the performance might suck. But even if this is not the case, my previous point stands, and I would personally go for Michal’s solution.

Using that approach, you could have something like:

defmodule Const do
  # Michal's snippet:
  values = [source_lvl1_api: 301023, ...]
  for {key, value} <- values do
    def encode(unquote(key)),   do: unquote(value)
    def decode(unquote(value)), do: unquote(key)
  end
end

And now you can do e.g. Const.encode(:source_lvl1_api), or Const.decode(301023) to perform atom <-> integer conversions.

So what Michal is trying to tell you is that as soon as you take some input from say HTTP request, or the database, you invoke Const.decode to convert it into an atom. Then in the rest of your code, you just deal with atoms (e.g. :source_lvl1_api), so the code is ridden of magical numbers. Likewise, if you need to send a response to some client, or store to the database, you perform Const.encode to convert the atom into a corresponding integer.

9 Likes

I ended up going with the idea from @michalmuskala - it makes sense to use atoms everywhere I can, then just have integers in the DB and frontend client.

defmodule API.Const do
  @root_dir       File.cwd!
  @consts_dir     Path.join(~w(#{@root_dir} lib api common constants))

  consts = Path.wildcard("#{@consts_dir}/**/*.json")
           |> Enum.map(fn(filename) -> File.read!(filename) |> Poison.decode! end)
           |> Enum.reduce(fn(c, map) -> Map.merge(map, c) end)

  for {key, value} <- consts do
    def encode(unquote(String.to_atom(key))),   do: unquote(value)
    def decode(unquote(value)), do: unquote(String.to_atom(key))
  end
end

I decided to have a folder of JSON files, that way I can easily use it on the frontend as well (at the moment it runs some scripts to convert to JSON files from PHP)

Are there any issues doing it this way?

I was thinking of prepending the name of the file to each key in the JSON file to help namespace it a bit, so system.json would be

{
  "SOURCE_LVL1_API": 301023,
  "SOURCE_LVL1_HTTP": 301024,
  "SOURCE_LVL1_EMAIL": 301025,
  "SOURCE_LVL1_SMS": 301026,
  "SOURCE_LVL1_UNKNOWN": 301027
}

and would end up being

%{
  :SYSTEM_SOURCE_LVL1_API => 301023,
  :SYSTEM_SOURCE_LVL1_HTTP => 301024,
  :SYSTEM_SOURCE_LVL1_EMAIL => 301025,
  :SYSTEM_SOURCE_LVL1_SMS => 301026,
  :SYSTEM_SOURCE_LVL1_UNKNOWN => 301027
}

Not sure though!

2 Likes

When using an external file like that, consider adding @external_resource path to the module - it will tell the elixir compiler to recompile the module if the file changes.

4 Likes

You could also reach for multiple “nested” modules. Basically, for each input json file name, create the module alias using Module.concat, and then define the module dynamically. That way you could invoke say Const.System.decode/encode, assuming the input file is system.json.

Thanks guys, I’m really liking where this has gone! So, I’ve tried to implement both of those ideas, and I now have

defmodule API.Const do
  @root_dir       File.cwd!
  @const_dir      Path.join(~w(#{@root_dir} lib imsapi common constants))
  @const_files    Path.wildcard("#{@const_dir}/**/*.json")

  for filename <- @const_files do
    # Watch for changes and recompile
    @external_resource filename

    # This will be the new module: API.Const.<group>
    group = filename
            |> Path.basename(".json")
            |> String.capitalize

    file_consts = File.read!(filename)
                  |> Poison.decode!

    defmodule Module.concat([API, Const, group]) do
      for {key, value} <- file_consts do
        def encode(unquote(String.to_atom(key))),   do: unquote(value)
        def decode(unquote(value)), do: unquote(String.to_atom(key))
      end
    end
  end
end

That makes a much nicer way to access the different files, so thanks for that suggestion @sasajuric! I’m not sure if I’m doing something wrong, but it doesn’t seem to recompile if I modify one of the JSON files?

I’m running it using iex -S mix phoenix.server

Seems I’ve been spending too much time using things that live reload code, the files DO recompile if I change the JSON, I was just waiting for it to happen live :blush:

I’m sure there are things I can tweak in the below, but it seems to work fine for what I need right now.

  • Split over multiple JSON files
  • Custom Ecto field for converting when entering/retrieving from DB
  • Allow atoms to be used during dev/debug, but integers for storage/comms
  • Re-compiles when any JSON file changes
  • Dynamically create “nested modules” based on the JSON filename

So this is end result of all the suggestions (thanks again!)

defmodule API.Const do
  @moduledoc """
  A single point that we can use constants from. Rather than using something
  like a map to expose atoms => integers we create a set of functions that 
  let us encode/decode from either atom => integer or integer => atom. Doing this
  makes it a lot easier to debug things as we are basically just using atoms 
  everywhere in the code - we change to integers on the app boundary.

  We still break up the constants with regards to their respective areas, and 
  then we just include everything in here. We store the constatnts as JSON files
  so that we can export into frontend clients if needed,

  Discussion here: https://elixirforum.com/t/shared-module-constants/2799/
  
  Eg:

  API.Const.System.encode(:ORG_STATUS_ACTIVE) => 100101
  API.Const.System.decode(100101) => :ORG_STATUS_ACTIVE
  """

  @root_dir       File.cwd!
  @const_dir      Path.join(~w(#{@root_dir} lib imsapi common constants))
  @const_files    Path.wildcard("#{@const_dir}/**/*.json")

  for filename <- @const_files do
    # Watch for changes and recompile
    @external_resource filename

    # This will be the new module: API.Const.<group>
    group = filename
            |> Path.basename(".json")
            |> String.capitalize

    file_consts = File.read!(filename)
                  |> Poison.decode!

    
    # Creates a module under the main API.Const module using the filename.
    # 
    # Eg a system.json file would end up as `API.Const.System`
    defmodule Module.concat([API, Const, group]) do
      for {key, value} <- file_consts do
        def encode(unquote(String.to_atom(key))),   do: unquote(value)
        def decode(unquote(value)), do: unquote(String.to_atom(key))
      end

      # Creates a new module that can be used with Ecto models to hsve s field
      # that is automatically encoded/decoded as data is retrieved/entered.
      # Name is the same as the outer module, with `Ecto` appended
      # 
      # Eg a system.json file would end up as `API.Const.System.EctoField`
      defmodule Module.concat([API, Const, group, EctoField]) do
        @behaviour Ecto.Type

        @group_mod Module.concat([API, Const, group])

        def type, do: :integer

        def cast(int) when is_integer(int), do: {:ok, apply(@group_mod, :decode, [int])}
        def cast(atom) when is_atom(atom), do: {:ok, atom}
        def cast(_), do: :error

        def load(int) when is_integer(int), do: {:ok, apply(@group_mod, :decode, [int])}
        def load(_), do: :error

        def dump(atom) when is_atom(atom), do: {:ok, apply(@group_mod, :encode, [atom])}
        def dump(_), do: :error
      end
    end
  end
end
2 Likes

Coming from a static language background it’s really hard to get around the “elixir-way”.

@OvermindDL1 What would be your approach if instead of integers you’d need a list of atoms?
Imagine if you needed to validate a list of valid countries; it seems a bit overkill (due to repetition) to just:

def countries, do: %{
united_states: :united_states,
canada: :canada,
paris: :paris,
england: :england
}

# maybe just... (???)
# def valid_countries, do: [:united_states, :canada, :paris, :england]

Instead of simply scattering atoms in the application, I like the idea to have a “single source of truth” to serve like self-documentation (mainly autocomplete). In that case, I’d typically use an enum or a class with static const properties in C#.

PS.: BTW I don’t like very much the idea of using loose JSON files in the project like @benperiton.

Just return a list of them then, no need to duplicate them. :slight_smile:

Making a map of them can still be quite useful because it allows for O(log N) lookup instead of O(N) if you just need to verify it is valid or not (I generally do the values as just true in that case because convention, but doesn’t really matter).

Likewise, and baking them into a module like this is the way to do that, the values all get interned into the system and no GC will touch them and their linked usage anywhere will not trigger GC either, it’s very efficient.

1 Like

I’m trying to achieve the same thing but with autocomplete functionality to know which atoms are available for use.
Would there be any performance drawbacks if I create a function for each key value pair?
I’m new to Elixir, sorry if this is a silly question

defmodule ErrorMessage do
  values = [
    account_not_found: "Error message for account not found" ,
    account_blocked: "Error message for account blocked" ,
    account_suspended: "Error message for account suspended",
    ...
  ]

  for {key, value} <- values do
    def unquote(:"#{key}")(), do: unquote(value)
  end
end

And then, when I need to use the error message

alias ErrorMessage, as: E
assert response === E.account_not_found

This way the editor offers me all available functions in the module as autocomplete

I recommend that you take a look at the EctoEnum library for insights.

It’s worth mentioning to people coming to this thread that after getting used to Elixir it feels more natural to pass atoms around, so that’s not a big deal. The minor drawback is refactoring; you’ll have to find-all/ replace-all instead of just hitting the refactor/ rename button in your editor (the outcome tends to be virtually the same).

In most cases, having good documentation on the available options is sufficient.
Also, I think you shouldn’t worry too much about having autocomplete for that. Instead, you can write documentation and typespecs for your public APIs to achieve a similar goal.

3 Likes