Elegant solution for constants?

Questions / Help
,
1

Hey there,

Most of the advice I see on something similar to contants in Elixir points me towards using techniques such as:

defmodule MyApp.Globals do

@one_value 1
@one_atm :one

def constant1(), do: 1234

def num(:one), do: 1
def num(:two), do: 2

def atm(1), do: :one
def atm(2), do: :two

end

But using those values in other modules is proving combersome, probably because I’m missing some key understanding of Elixir.

AFAIK I cannot use functions in pattern match clauses so I cannot write

def some_func(%Somestruct{tag: MyApp.Globals.num(:one)}) do...

Also, when I need the values in an ecto query I have to “load” them into variables first, like:

val = MyApp.Globals.num(:two)
...
query...where: a.tag == ^val
...

I’ve considered Ecto Enums but I needed more control and error checking because the use-case is a crude form of polymorphism and I wish to ensure that at compile-time I can catch duplicate values and names.

I’m really hoping to keep all these definitions in one place where I can assert such controls because they are essential for the correct operation of my application, but the eloborate database mechanism I used before was just more of a nuisance than anything else.

My objective is to have there symbolic values defined in one place but when I use them they are the exact equivalents of actual numeric constants and atoms.

How can I go about that?

P.S. My ancient C-mindset is probably my worst enemy here, as it was so common to have a header file defining constants with #defines.

3 Likes
2

You can use macros then

defmacro __some_constant do
  quote do
    "stuff"
  end
end

You can use it in pattern maches. Double undescore denotes the macro type, it’s just a convention.

1 Like
3

Elixir does not have constants, but it’s possible to do what you need with a code like:

defmodule MyLib do
  defmacro const({:=, _meta, [{var_name, _var_meta, _var_module}, right]}) do
    quote bind_quoted: [name: var_name, value: right] do
      defmacro constant(unquote(name)) do
        Macro.escape(unquote(value))
      end
    end
  end

  defmacro constp({:=, _meta, [{var_name, _var_meta, _var_module}, right]}) do
    quote bind_quoted: [name: var_name, value: right] do
      defmacrop constantp(unquote(name)) do
        Macro.escape(unquote(value))
      end
    end
  end
end

defmodule MyApp do
  import MyLib

  const x = 5

  def example(x = constant(:x)), do: {:ok, x}
  def example(x), do: {:error, "expected x to be #{constant(:x)} got #{inspect(x)}"}
end

Please keep in mind that for a custom DSL you have to add it to the .formatter.exs in order to not add parentheses.

macro_dsl = [const: 1, constp: 1]

[
  export: [locals_without_parens: macro_dsl],
  locals_without_parens: macro_dsl,
  # …
]

With such configuration your project and every project depending on it would format your DSL correctly. Also remember to use import_deps in project depending on your one to make sure your formatter configuration is used.

[
  import_deps: [:your_app_name],
  # …
]
7 Likes
4

Give @sasajuric 's method a try.

IMO, it’s simple and powerful.

You can also try some packages, like GitHub - jcomellas/ex_const: Constants and Enumerated Values for Elixir

2 Likes
5

I’ve used this method to share constants within a few modules of a context once or twice. It’s not perfect as the middle attributes appear out of nowhere (well, the use Some.Constants call) but then you can use them in guards and pattern matching with plain old elixir syntax.

6

Do unused module attributes get thrown away? IE, do you have to be careful with this solution to not make one massive module of constants where some could be potentially very large (for one reason or another)? Or does the compiler take care of this?

7

I don’t know. I thought there value of each module attribute got inserted at each use site. That’s one argument people make against this method when pitching the function method.

8

That I know they do, though not a big deal if everything is small. After asking I realized I could just play around and find out and it looks like they are indeed thrown away.

1 Like
9

Module attributes are inlined where they’re used unless Module.register_attribute(module, attr, persist: true) is called at compile-time. (In which case they’re available using module.__info__(:attributes), iirc.)

I don’t think “overly-large module” is a practical concern unless you’re talking an extremely large number of constants, at which point you could consider persisting that data to a separate file, using @external_resource to “link” it to the module (making it a dependency), or reading it in (and potentially cache parts of it) at runtime.

2 Likes
10

I edited my response, but to clarify, @external_resource isn’t needed if the file in question is only read at runtime. It’s necessary if the file is a compile-time dependency (e.g. you generate functions from it), as it forces the module to recompile if the file changes.

1 Like
11

I shall look into that.

Your’s is a little more complicated but I’m sure once I found a way to wrap my head around what it’s really doing I will appreciate it more fully.

Meaning @michalmuskala’s I presume. Yes, that seems simple and powerful.

Actually I’ve already taken to using macros in what I’ve done so far, just not in that manner. I wrote a macro that took a map as input parameter and then performed the duplicate checks compile time before defining the functions other sources lead me to believe is as close as one can get to symbolic constants in Elixir.

I shall now go play with the examples to see if I can get it to work for my use case without getting entangled in nested macros because I don’t fully understand what it is doing.

I’ll check back when I get stuck. Thank you all for the pointers.

2 Likes
12

I managed to implement it in many different ways but couldn’t wrap my head around module attributes well enough to have something that I only ever specify in one place, will check for duplicates at compile time and that I can also use as a stable source for Ecto.Enum values to make these a bit more useful in practice.

In the end, I found SimpleEnum to do what I needed.

I found that for the results I wanted (describerd as Fast access by SimpleEmum there is no run-time performance hit for large definitions or small ones. It’s all done at compile time so a large file might slow your compiles down a tad when you’ve changed the definitions but the impact at run-time is negligible.

Thanks for caring and all the advice.

P.S. @Eiji your feedback regarding DSLs went well over my head. Thanks anyway.

1 Like
13

First of all I made a small mistake in my examples, so quote calls:

    quote bind_quoted: [name: var_name, value: right] do

should be:

    quote bind_quoted: [name: var_name, value: Macro.escape(right)] do

With Macro.escape/1 call in each macro we are able to escape AST of non-literals, so the “constants” here are able to store more complex structs like a maps. :bulb:

If you are not good with attributes you can use Process module to set and get data which would be stored in process dictionary. Having map as value in said dictionary together with is_map_key/2 guard should be more than enough to handle duplicates easily. :+1:

However you should pay attention to a place in code where you want to work on. For example inside const macro (as you can see in pattern-matching) the argument passed to macro are changed to their AST form. On the other side inside generated constant macro we are no longer sure when said generated macro would be called and how many times. So the golden spot is inside quote’s do … end block, but outside of any function or macro generation. :thinking:

defmacro generates_macros(…) do
  # arguments are in abstract form
  quote … do
    # golden spot, we are still in compile-time,
    # but the arguments here are no longer in AST form (unless previously escaped)
    defmacro macro_to_be_generated(…) do
      # arguments are in abstract form
      # we are not sure when said macro would be called
    end
  end
end

Of course you can work also outside of quote call, but it’s easiest if you then assume that each argument passed is a literal value and not a variable. :see_no_evil:

My example was to show that DSL does not need to have a hundreds of lines. They are relatively short and easy to understand (sa long as you understand basics of metaprogramming). It’s great that you have found a library that would do that for you, but this still should be interesting for a learning purposes. :books:

Compilation time is not so important especially when on the second side of the coin is a requirement to deal with a possible bottlenecks (like a concurrent file access). Over a time you would be used to write macros every time you see a way to optimise runtime. It’s nothing to do with a simple scripts or a hobby projects, but a smallest change of time per operation may become very important at scale. :chart_with_upwards_trend:

14

I disagree. Compile time is important. The faster you can compile, the faster the feedback loop can be during development. If the compile time suffers while the runtime difference is small, you are over optimizing as long as you do not hit the bottleneck. Not to mention the downside of (many) macro’s in a codebase.

That being said: I currently work on a project which does rely on programmatically generating code just to save cycles during runtime. So as with everything: it depends.

2 Likes
15

Maybe it’s just me, but I’m always surprised seeing comments like that. Developers should not work on “netbooks with low DPI monitor” or similar hardware (again lots of comments in different topics). Above 10 years ago I was compiling Gentoo with KDE, LibreOffice and Firefox - all of it took me around a week (first time using Linux + compilation fails at night). At that time any kind of improvement was a truly life saver. That happen when I was right after finishing high school. I did not had any funds and so on … :tired_face:

Now I’m a senior developer who bough a little monster for work on tiny 2-week long project. While I would disagree that anyone on well-paid position should have “everything” then we are talking about hardware we work on daily, so for me investing in it is maybe not a top 1 priority, but it’s absolutely very important. Something that 10 years ago as a hobby took me terribly long, but now I can do within a few hours … in a virtual machine! :zap:

Since not every project is a “web browser or operating system” the compilation time in my personal opinion should be a negligible parameter especially if lots of community members are using Apple’s hardware. That’s said … compilation as a whole process have indeed a very big impact on feedback. If preparing environment is too complicated or documentation mentions only docker then I’m not surprised that some people may be less interested in contribution. :thinking:

That’s why I absolutely love Mix.install feature! Honestly I was never checking how many dependencies it has to fetch and compile. Many times I was using it just to see a phoenix documentation in console. So again maybe it’s only crazy me, but since programming was not just a hobby I absolutely never considered a compilation time important at all. :heart:

For sure in every topic going on one of the edges is a terrible idea with even worse consequences in long term. You may be surprised, but I’m much closer to the centre than you think. I’m rather a slightly more on macros side, just because I take much more care on runtime than compile time. :see_no_evil:

I often decide to use metaprogramming to for example generate a pattern-matching based on non-user input that in normal case would need to be done on runtime. On contrary when writing scripts I’m rather not writing macros. :bulb:

Fully agree, that’s why many years ago I decided to write posts on forums more generally. My favourite quote is:

There is not a single rule, however plausible, and however firmly grounded in epistemology, that is not violated at some time or other. It becomes evident that such violations are not accidental events, they are not results of insufficient knowledge or of inattention which might have been avoided. On the contrary, we see that they are necessary for progress.

Paul Feyerabend

:+1:

16

I don’t think anybody cares about how fast it compiles on your local machine, but when it comes to CI, where you can easily run 20 pipelines for a single commit, having that time as low as possible is very important.

1 Like
17

Oh, that makes sense. I was working on matrixes and so on, but definitely not on so much. Well … if it’s about me I’m checking all stuff locally (credo, dialyzer and tests) before commit, so I’m not waiting for a results from CI no matter how long it would take. I also don’t believe that project maintainers or other contributors have nothing more interesting than waiting for CI checks of my work. So … I see your point, but I’m used to async work so things like CI checks are completely not important for me (in context of time they need of course). :see_no_evil:

Personally I recommend to install this:

and do something else even if it’s about watching videos with funny cats. :joy_cat:

btw. Since tests calls runtime code even few times (depending on case) isn’t moving things to compile time still an improvement of CI checks time? I never tested that, so I’m not sure … :thinking:

18

Forgive my poor word choices that sparked such a debate about compile time. The sentiment I tried to express was indeed that compile time performance is not a big deal even though it will be impacted if and when you make changes to the definitions. Lots of source, whether the source is the result of macro expansion or, produced by AI, copied and pasted or written the old fashioned way will always take time to compile. For the cycle-time of the developer’s feedback loop the hope is that only changes would trigger a compile, but it would be interesting to know the facts about how used, imported and required modules plays at that. If you had a gigantic symbol/enum defining module which is used by many modules where the using macro requires and aliases the big module, will the time to compile/process every such module that uses the big one be impacted as well?

19

As above it depends … :thinking:

Let’s take at very simple example:

Mix.install([:benchee])

defmodule Example do
  @map %{a: 5, b: 10, c: 15}

  for {key, value} <- @map do
    def compile_time(unquote(key)), do: unquote(value)
  end

  def run_time(key) do
    Map.fetch!(%{a: 5, b: 10, c: 15}, key)
  end
end

Benchee.run(%{
  "compile time" => fn -> Example.compile_time(:b) end,
  "run time" => fn -> Example.run_time(:b) end
})

So there is improvement, but is it worth? Well … If it’s your script that you use at most once a day then you save around 2ns. That’s why I wrote that usually I’m not writing macros in such cases. :bulb:

However at scale we have around 10% improvement. If same would happen in important part of your app or service then it means you have significantly improve UX (faster replies) or you are able to support 10% more clients at a time. If the numbers would be same then you could have extra 4M clients working at the same time without performance penalty. :tada:

Does it mean that macros are good only for big projects or projects that in production have millions of users? Definitely no. There are cases where you need to deal with lots of data alone. Surprisingly it happens more often than you think. A good example here are … web scrapers! You parse a huge amount of text as HTML file (sometimes you also would like to parse CSS and JavaScript). If there is an improvement in parser like the one in example above then we have a big improvement already, but it’s not all. :open_file_folder:

Often when writing scrapers you have to deal with a list of links and / or pagination, so you may want to parse hundreds or even thousands of pages. Alone it’s still not much, but you may start to feel extra time counted in seconds. If you made such a change for learning purposes it was definitely worth, but otherwise you simply have other things to do than looking for a “possible improvement”. That’s said … if you take a look at your code after few months or especially after years then you would easily find that some things you can immediately find and fix. :toolbox:

So there are cases where any optimisation generally are not worth, but sometimes they may be even expected. That’s how we’re back to the start: “it depends”. :see_no_evil:

I never worked on anything “gigantic”. Think about Phoenix contexts. As same as you do not put all contexts into single file as same you most likely would not put all enums and related logic to one file. There are few reasons for it:

  1. Readability i.e. too many lines of code
  2. Naming problems, so if lots of enums or constants would have lots of macros or other functions imported then it would be hard to find a naming that does not conflicts (in terms of readability) with any imports
  3. Warnings - Elixir would warn when a module compiles longer than 10s - that’s very important hint for a refactor
  4. Content - I have no idea what you want to put there, but dozens of hundreds of imported functions most probably does not affect the compilation time too much.

There are most probably many other reasons, but this should be more than enough, so sooner or later you would end up with splitting all of that into smaller pieces and then it then it would be never a problem. This is rather a theory as in practice you would rather not end up with a gigantic single file.

20

More appologies if I left you with the impression that I was concerned with compile time vs run time resolution of constants because I’m not.

I only use the “compile time” form and my set of definitions will probably remain ~ < 100 elements forever.

In the discourse someone warned against (possible) performance impact for very large definitions using the techniques under discussion but framed it in a manner which assumed run time performance. I merely pointed out that by sticking to compile time resolution the run time performance is already optimal.

Then someone mentioned scenarios where compile time performance does become significant.

Which I acknowledged may be the case (as a side-issue) and wondered in principle what the impact of large defininition modules would have on compile time.

Specifically, I wondered if a module using such macros as we’ve been talking about gets compiled once and simply loaded by the compiler every time it gets used in another module, or whether the compiler would have to reprocess/recompile the module as part of every other module that uses it?