I’m porting an old PHP application over to Elixir and Phoenix, and I’m struggling to figure out the best way to transfer the constants that currently exist.
At the moment there are multiple files with various constants used throughout the app, for instance:
<?php
use Shared\Const\System as SysConst;
if (SysConst::SOURCE_LVL1_API === $someOtherVariable)
{
echo 'Starts at API';
}
Is there a way to do something similar in Elixir? Or is there a better way to store lots of constants that could be the same name, but relate to different areas?
The usual approach would be to use atoms inside the system, e.g. :source_lvl1_api and convert to/from integer encoding on the system boundary, if needed. This makes it easy for debugging and introspection since at runtime you have readable atoms, instead of opaque integer values flying around. You can generate the conversion functions easily with a sprinkle of macros:
values = [source_lvl1_api: 301023, ...]
for {key, value} <- values do
def encode(unquote(key)), do: unquote(value)
def decode(unquote(value)), do: unquote(key)
end
Being a constant object in the code with no variables it gets constructed fast, however usual things in Erlang/Elixir get optimized into the bytecode so no construction actually takes place and it gets shared easily, I’m not sure if maps do that however.
Overall @michalmuskala method is usually best, no construction time at all, however the lookup time can be linear in a match set like that so a map might be faster in lookup speed.
So it looks like @OvermindDL1 idea of using a map would be the easiest way to get them across, but I’m intrigued by what you said @michalmuskala.
I’m not sure I fully grasp what you are suggesting, are you saying that the conversion would happen at the point that things are inserted and retrieved from the database? I’m struggling to figure out where those macros would live and how they would be used in other modules.
I have a macro for reusing constants across multiple modules. The benefit being that they work in matches and in guards.
defmodule Constants do
@moduledoc """
An alternative to use @constant_name value approach to defined reusable
constants in elixir.
This module offers an approach to define these in a
module that can be shared with other modules. They are implemented with
macros so they can be used in guards and matches
## Examples:
Create a module to define your shared constants
defmodule MyConstants do
use Constants
define something, 10
define another, 20
end
Use the constants
defmodule MyModule do
require MyConstants
alias MyConstants, as: Const
def myfunc(item) when item == Const.something, do: Const.something + 5
def myfunc(item) when item == Const.another, do: Const.another
end
"""
defmacro __using__(_opts) do
quote do
import Constants
end
end
@doc "Define a constant"
defmacro constant(name, value) do
quote do
defmacro unquote(name), do: unquote(value)
end
end
@doc "Define a constant. An alias for constant"
defmacro define(name, value) do
quote do
constant unquote(name), unquote(value)
end
end
end
This is quite cool! Would be nice if it was possible to require and alias at the same time, but I think I might have a play with this as the general way to do constants.
Well, in the “required” module you could implement a __using__/1 macro and use it from the requiring one. In that __using__/1 you then can of course require and alias at the same time.
Using such macros has still the same issue - you have opaque integers passing around at runtime - this makes it really hard to debug issues, especially if dealing with large numbers.
As to when do conversions - I spoke about boundaries of the system. By that I mean every time we receive data from outside or send it out - this means params from requests, database responses, etc. For example, when using ecto, you can provide a custom type to do the conversions for you (assuming encode and decode functions as before):
defmodule Constant do
@behaviour Ecto.Type
def type, do: :integer
def cast(int) when is_integer(int), do: {:ok, decode(value)}
def cast(atom) when is_atom(atom), do: {:ok, atom)
def cast(_), do: :error
def load(int) when is_integer(int), do: {:ok, decode(value)}
def load(_), do: :error
def dump(atom) when is_atom(atom), do: {:ok, encode(value)}
def dump(_), do: :error
end
I would definitely go for the approach from @michalmuskala. One problem with maps is that it only supports one way conversion (converting a const into an integer). If you need to convert an integer into a const, you’ll have to build a reversal map. This can be easily done, even during compilation time, but then the code becomes as complex if not more than Michal’s version.
Moreover, I’m not completely sure whether this map is stored in the so called “constant pool”, and if it’s not, then the performance might suck. But even if this is not the case, my previous point stands, and I would personally go for Michal’s solution.
Using that approach, you could have something like:
defmodule Const do
# Michal's snippet:
values = [source_lvl1_api: 301023, ...]
for {key, value} <- values do
def encode(unquote(key)), do: unquote(value)
def decode(unquote(value)), do: unquote(key)
end
end
And now you can do e.g. Const.encode(:source_lvl1_api), or Const.decode(301023) to perform atom ↔ integer conversions.
So what Michal is trying to tell you is that as soon as you take some input from say HTTP request, or the database, you invoke Const.decode to convert it into an atom. Then in the rest of your code, you just deal with atoms (e.g. :source_lvl1_api), so the code is ridden of magical numbers. Likewise, if you need to send a response to some client, or store to the database, you perform Const.encode to convert the atom into a corresponding integer.
I ended up going with the idea from @michalmuskala - it makes sense to use atoms everywhere I can, then just have integers in the DB and frontend client.
defmodule API.Const do
@root_dir File.cwd!
@consts_dir Path.join(~w(#{@root_dir} lib api common constants))
consts = Path.wildcard("#{@consts_dir}/**/*.json")
|> Enum.map(fn(filename) -> File.read!(filename) |> Poison.decode! end)
|> Enum.reduce(fn(c, map) -> Map.merge(map, c) end)
for {key, value} <- consts do
def encode(unquote(String.to_atom(key))), do: unquote(value)
def decode(unquote(value)), do: unquote(String.to_atom(key))
end
end
I decided to have a folder of JSON files, that way I can easily use it on the frontend as well (at the moment it runs some scripts to convert to JSON files from PHP)
Are there any issues doing it this way?
I was thinking of prepending the name of the file to each key in the JSON file to help namespace it a bit, so system.json would be
When using an external file like that, consider adding @external_resource path to the module - it will tell the elixir compiler to recompile the module if the file changes.
You could also reach for multiple “nested” modules. Basically, for each input json file name, create the module alias using Module.concat, and then define the module dynamically. That way you could invoke say Const.System.decode/encode, assuming the input file is system.json.
Thanks guys, I’m really liking where this has gone! So, I’ve tried to implement both of those ideas, and I now have
defmodule API.Const do
@root_dir File.cwd!
@const_dir Path.join(~w(#{@root_dir} lib imsapi common constants))
@const_files Path.wildcard("#{@const_dir}/**/*.json")
for filename <- @const_files do
# Watch for changes and recompile
@external_resource filename
# This will be the new module: API.Const.<group>
group = filename
|> Path.basename(".json")
|> String.capitalize
file_consts = File.read!(filename)
|> Poison.decode!
defmodule Module.concat([API, Const, group]) do
for {key, value} <- file_consts do
def encode(unquote(String.to_atom(key))), do: unquote(value)
def decode(unquote(value)), do: unquote(String.to_atom(key))
end
end
end
end
That makes a much nicer way to access the different files, so thanks for that suggestion @sasajuric! I’m not sure if I’m doing something wrong, but it doesn’t seem to recompile if I modify one of the JSON files?
Seems I’ve been spending too much time using things that live reload code, the files DO recompile if I change the JSON, I was just waiting for it to happen live
I’m sure there are things I can tweak in the below, but it seems to work fine for what I need right now.
Split over multiple JSON files
Custom Ecto field for converting when entering/retrieving from DB
Allow atoms to be used during dev/debug, but integers for storage/comms
Re-compiles when any JSON file changes
Dynamically create “nested modules” based on the JSON filename
So this is end result of all the suggestions (thanks again!)
defmodule API.Const do
@moduledoc """
A single point that we can use constants from. Rather than using something
like a map to expose atoms => integers we create a set of functions that
let us encode/decode from either atom => integer or integer => atom. Doing this
makes it a lot easier to debug things as we are basically just using atoms
everywhere in the code - we change to integers on the app boundary.
We still break up the constants with regards to their respective areas, and
then we just include everything in here. We store the constatnts as JSON files
so that we can export into frontend clients if needed,
Discussion here: https://elixirforum.com/t/shared-module-constants/2799/
Eg:
API.Const.System.encode(:ORG_STATUS_ACTIVE) => 100101
API.Const.System.decode(100101) => :ORG_STATUS_ACTIVE
"""
@root_dir File.cwd!
@const_dir Path.join(~w(#{@root_dir} lib imsapi common constants))
@const_files Path.wildcard("#{@const_dir}/**/*.json")
for filename <- @const_files do
# Watch for changes and recompile
@external_resource filename
# This will be the new module: API.Const.<group>
group = filename
|> Path.basename(".json")
|> String.capitalize
file_consts = File.read!(filename)
|> Poison.decode!
# Creates a module under the main API.Const module using the filename.
#
# Eg a system.json file would end up as `API.Const.System`
defmodule Module.concat([API, Const, group]) do
for {key, value} <- file_consts do
def encode(unquote(String.to_atom(key))), do: unquote(value)
def decode(unquote(value)), do: unquote(String.to_atom(key))
end
# Creates a new module that can be used with Ecto models to hsve s field
# that is automatically encoded/decoded as data is retrieved/entered.
# Name is the same as the outer module, with `Ecto` appended
#
# Eg a system.json file would end up as `API.Const.System.EctoField`
defmodule Module.concat([API, Const, group, EctoField]) do
@behaviour Ecto.Type
@group_mod Module.concat([API, Const, group])
def type, do: :integer
def cast(int) when is_integer(int), do: {:ok, apply(@group_mod, :decode, [int])}
def cast(atom) when is_atom(atom), do: {:ok, atom}
def cast(_), do: :error
def load(int) when is_integer(int), do: {:ok, apply(@group_mod, :decode, [int])}
def load(_), do: :error
def dump(atom) when is_atom(atom), do: {:ok, apply(@group_mod, :encode, [atom])}
def dump(_), do: :error
end
end
end
end
Coming from a static language background it’s really hard to get around the “elixir-way”.
@OvermindDL1 What would be your approach if instead of integers you’d need a list of atoms?
Imagine if you needed to validate a list of valid countries; it seems a bit overkill (due to repetition) to just:
Instead of simply scattering atoms in the application, I like the idea to have a “single source of truth” to serve like self-documentation (mainly autocomplete). In that case, I’d typically use an enum or a class with static const properties in C#.
PS.: BTW I don’t like very much the idea of using loose JSON files in the project like @benperiton.
Just return a list of them then, no need to duplicate them.
Making a map of them can still be quite useful because it allows for O(log N) lookup instead of O(N) if you just need to verify it is valid or not (I generally do the values as just true in that case because convention, but doesn’t really matter).
Likewise, and baking them into a module like this is the way to do that, the values all get interned into the system and no GC will touch them and their linked usage anywhere will not trigger GC either, it’s very efficient.
I’m trying to achieve the same thing but with autocomplete functionality to know which atoms are available for use.
Would there be any performance drawbacks if I create a function for each key value pair?
I’m new to Elixir, sorry if this is a silly question
defmodule ErrorMessage do
values = [
account_not_found: "Error message for account not found" ,
account_blocked: "Error message for account blocked" ,
account_suspended: "Error message for account suspended",
...
]
for {key, value} <- values do
def unquote(:"#{key}")(), do: unquote(value)
end
end
And then, when I need to use the error message
alias ErrorMessage, as: E
assert response === E.account_not_found
This way the editor offers me all available functions in the module as autocomplete
I recommend that you take a look at the EctoEnum library for insights.
It’s worth mentioning to people coming to this thread that after getting used to Elixir it feels more natural to pass atoms around, so that’s not a big deal. The minor drawback is refactoring; you’ll have to find-all/ replace-all instead of just hitting the refactor/ rename button in your editor (the outcome tends to be virtually the same).
In most cases, having good documentation on the available options is sufficient.
Also, I think you shouldn’t worry too much about having autocomplete for that. Instead, you can write documentation and typespecs for your public APIs to achieve a similar goal.
