Easiest way to "step system time forward" in tests to test expiration type effects?

If Elixir was a more traditional object oriented language, I would probably do something simple like this:

static class DateManager {
    public static timeOffsetSeconds = 0;

#if DEVELOPMENT
    public static getTimeSeconds(){
        return System.system_time(:second) + timeOffsetSeconds;
    }
#else
    public static getTimeSeconds(){
        return System.system_time(:second);
    }
#endif

}

Then in a development build, when one calls the following code, it is easy to simulate time moving forward for all modules that rely on the getTimeSeconds() function:

runInitialFunction(); //depends internally on  DateManager.getTimeSeconds()
DateManager.timeOffsetSeconds += 60*60; //move "system time" forward an hour
runSecondFunction(); //depends internally on DateManager.getTimeSeconds()

Compiler flags like above will even omit the “offset” function so it can’t be theoretically hijacked or abused in any way (won’t exist in real system).

We don’t have any easy manner of “state” however in Elixir without starting a GenServer or cache like ets or mnesia. I don’t want to get into the habit of starting random otherwise empty GenServers/ets/mnesia stores just so I can have “state” based functions like this. Not likely correct and while not terribly wasteful, I doubt this is intended.

Though it does seem possible as a simple hack. One can create one GenServer or ets entry for each “static” class that must hold some state on Application startup and register the PID or know the ets entry name. Perhaps simplest would be to make an ets entry for time offset. But then we don’t have compiler flags so in the real development you are wasting time going to the ets to check for a 0 value offset constantly.

Without re-writing every function everywhere to take a time_offset input variable (which seems like also bad practice), what would be the simplest way to accomplish the above?

If I have something like the following that my modules are using to get time:

defmodule TimeManager do
    def get_time_seconds() do
         System.system_time(:second)
    end
end

Is there any easy and safe way to introduce a step forward effect to it as needed in my tests?

The Mimic and Repatch libraries allow you to mock f.ex. the DateTime module functions.

2 Likes

Thanks. I have looked into the Mimic package but it only seems to replace the function when it is called from inside the test block. This does not help the situation where every function in the system is calling say DateManager.get_time_s() internally and we want to replace that function globally over the course of the test function run.

For example, I tried:

defmodule TokenTest do
    use ExUnit.Case
    import Mimic

    setup :set_mimic_global
    setup :verify_on_exit!


    test "time expiration token" do
        token = Token.create_token("name", "id", 1); #1 hour duration

        IO.puts("GET TIME PRE MIMIC: " <> inspect (TimeManager.get_time_s()))

        # move time forward
        TimeManager |> expect(:get_time_s, fn -> System.system_time(:second) + 60 * 60 + 1  end)

        IO.puts("GET TIME POST MIMIC: " <> inspect (TimeManager.get_time_s()))

        # test
        token_claims = Token.verify_token(token)
        user_name = Map.get(token_claims, "user_name")
        session_id = Map.get(token_claims, "session_id")
        correct = user_name == "name" && session_id == "id"
        assert correct == true

    end

This confirms that the time was moved forward in the test function. However, this still passes authentication, as it is not replacing the TimeManager.get_time_s() that is running inside Token.verify_token(token). This can be proven by outputting the value of TimeManager.get_time_s() from inside this verify function with IO.puts.

Is there any easy way to override a function like this for all functions that use it during the test?

How do people manage such situations?

I haven’t used it personally yet but I’d look into something like Klotho usage — klotho v0.1.2. There is a limitation section that’s helpful and you can always take inspiration to go your own path.

Use config to specify the module to use.

In test.ex: time_mod: MyTimeMod
In config.ex: time_mod: DateTime

Now you can fetch the module to use in code. You do have to mock function calls.

Update: https://rajrajhans.com/2023/01/writing-testable-elixir-code/

Ps. I use it for Ecto Repo calls with RepoWrite(able) and RepoRead(Only). In tests environment they are the same, in dev and prod they differ.

1 Like

Getting system time is a side-effect, so one direction you could go is getting rid of it in your functions and pass timestamp as an argument instead. Another way is mocking it. I recommend Mox — Mox v1.2.0. Repatch is also great, but using Mox is a good learning experience.

4 Likes

Wow. Sounds like there is no real good solution.

I am leaning towards combining this with my original idea. I can make two modules like this:

In test.ex: time_mod: TimeManagerTest
In config.ex: time_mod: TimeManager

Then in TimeManagerTest I can emulate the static variable with an :ets table entry to store and edit the time shift needed:

defmodule TimeManagerTest do
    defp add_table_if_needed() do

        exists = try do
            # Attempt to get info about the ETS table
            :ets.info(:time_offset_ets)
            true  # If no error occurs, the table exists
        catch
            :error, _ -> false  # If an error occurs, the table does not exist
        end
        if (!exists) do
            table = :ets.new(:time_offset_ets, [:set, :public, :named_table])
            :ets.insert(table, {:time_offset, 0})
        end
    end

    def set_time_shift_seconds(sec) do
        add_table_if_needed()
        :ets.insert(table, {:time_offset, sec})
    end

    def get_time_seconds() do
         add_table_if_needed()
         offset_val = :ets.lookup(:time_offset_ets, :time_offset)  
         offset_val = case (offset_val) do
              [{key, value}] ->
                    value
              _-> 0
         System.system_time(:second) + offset_val
    end
end

With this solution, I can then change time throughout my entire system during tests without having to rewrite anything else.

If you need static mutable variables especially in test functions it seems ets is the best way to do it. At least based on suggestions so far.

One should not want to rewrite all code that accesses “time” or the arguments they take just to manipulate time. This is probably the best bad solution. In theory this approach is not much different than having a static class in object oriented programming.

You absolutely don’t need ETS. Look at Mox, especially if your goto is env checks in preprocessor statements.

3 Likes

I don’t see how that will change the fact that I want to store an offset amount of time in the system and have it be applied everywhere in the system and then manipulate that offset between functions and still have it be represented everywhere in the system after manipulation.

We need some state storage. ets seems like the simplest way to store and manipulate static style state.

From what I see, just in terms of setting up the differential usage, I can declare the time module in config.exs:

config :my_app, :time_mod, TimeManager

Then when using this in my modules that rely on time I can do:

  @time_mod Application.get_env(:my_app, :time_mod)

And functions throughout my app will be:

current_time = @time_mod.get_time_seconds()

Then in tests I can override this with:

  setup do
    # Override the time module with the mock for tests
    Application.put_env(:my_app, :time_mod, TimeManagerTest)
    :ok
  end

Is that how you override the module roughly in the tests?

If so this should allow me to shift time with ease from there out in my tests.

I will try that.

Just for the record, this approach didn’t work. The @time_mod type references used by a module are set at compile. So they won’t be overridden by the setup function of the test trying to put_env.

In config/config.exs this worked:

config :main_server, :time_mod, TimeManager 

if config_env() == :test do
    config :main_server, :time_mod, TimeManagerTest
end

This truly overrides the module so when other modules are compiled containing @time_mod Application.get_env(:my_app, :time_mod) they get the right one. And I can just use that line at the start of each module that requires time. And use @time_mod thereafter.

This meets my criteria:

  • allows easy time shifting throughout the whole app in tests
  • no extra garbage code or vulnerabilities in real builds
  • no major redesign for any code needing time values

Easiest solution I think to all this and does seem to work.

1 Like

I think you proposed the best approach, second only to passing as an argument a function which returns the time.

We use a functional language, lets use our own patterns.

1 Like

I wanted to make override TimeManagerTest override TimeManager but this is very complex to do in practice.

I was thinking this would make a more general solution to the “need custom behavior from one function in a module on test cases” problem. Ie. If we can make a FileUploader module and then a FileUploaderTest module that otherwise inherits all its functions but we can override one function in it during tests.

This is suggested to work as an “Extension”.

I wonder about a system to inherit everything from the base class and then override a selected function. Any thoughts, even theoretically?

defmodule Extension do
  defmacro extends(module) do
    module = Macro.expand(module, __CALLER__)
    functions = module.__info__(:functions)
    signatures = Enum.map functions, fn { name, arity } ->
      args = if arity == 0 do
               []
             else
               Enum.map 1 .. arity, fn(i) ->
                 { String.to_atom(<< ?x, ?A + i - 1 >>), [], nil }
               end
             end
      { name, [], args }
    end

    zipped = List.zip([signatures, functions])
    for sig_func <- zipped do
      quote do
        defdelegate unquote(elem(sig_func, 0)), to: unquote(module)
        defoverridable unquote([elem(sig_func, 1)])
      end
    end
  end
end

Kind of wild. Anyone ever try something like this before?

The more I think about it, the more I also think :ets should be conceptually thought of as equivalent to a static class variable in any other language. A static variable is just a random piece of memory that everyone in the system can read to or write to. This is no different conceptually than an :ets table entry then. I am not sure if one would be faster than the other. But probably I would guess similar even there.

The reason concepts of inheritance, overriding, and static mutable state variables are so popular in coding languages is they solve common issues in a fast and easy way. Seems we can do the same here, though it is not popular.

It am surprised anyone would think a better solution would be to re-write every function that references time to take an otherwise extraneous time_offset variable just to facilitate some tests. In my opinion, that should be the absolute LAST option.

It also doesn’t make sense to me that we should have to rely on external libraries each with their own bizarre/unique/complex internal syntax and logic and maintainers that will just need another learning curve and risk for obsolescence when the tools are already freely available to do it in plain Elixir.

I don’t say any of that to be combative. As you said @krasenyp we can all find our own solutions. I just think it’s interesting to see how other people and people from different languages think differently.

Discussed in Change current time of EVM in tests - #3 by benwilson512

1 Like

As I said, I have no idea why you would want to do that. No offense intended, sincerely, and sorry if I sound blunt about it. This is just my opinion/feeling.

To do this, you are then rewriting your entire server code base and function architecture with pointless variable passing to accommodate the side case of some basic mock testing?

It could work, but that seems to me like the absolute worst solution possible. The ets solution was way easier and required no changes to the production code in any meaningful way.

Changing all your functions that use time to accommodate an offset and passing time by variables only seems like a good idea to me if you are religiously/stylistically committed to the idea of not persisting any variable-style state, even during a simple test, and even if it comes at your own detriment.

import DateTime, except: [function_x: 2]
def function_x(a,b), do: :bla

Thanks but I don’t think that exactly does what I wanted. What I wanted was this. But that doesn’t work. And I think your suggestion will have the same limitation. Ie. Import doesn’t copy all functions into the class. It just allows you to refer to the functions in that module without the full module name.

I did find a solution for true class inheritance here it seems.

Which provides a much better pattern I think for the problem raised in this thread overall.

ie. You can create Module and ModuleTest when you have something you want to override in Module during tests. Configure config.exs to use one or the other depending on environment into the env.

ModuleTest can inherit everything perfectly from Module it seems as long as Module’s functions are all public. And you can override any specific functions in ModuleTest as needed during simulation. For example, if there is a Module function called upload_file you can override it with a simulation in ModuleTest

This allows flexible mocking/testing code with no extra packages/dependencies.

This is a separate matter from the question of how to persist state (like the time offset using ets as I believe is most correct), but it gives a potentially good alternative option for the “override a function from a module during testing” issue.

Having played with Mimic a bit, I like this solution a lot more. Especially because you are now overriding the functions GLOBALLY throughout the code base for any function that calls them. Which Mimic does not do.

It’s also the reason for numerous bugs.

3 Likes

You seem to apply constructs of Object Oriented Programming onto Functional Programming. This leads to hard work, as if you would drive a screw in with a hammer.

I advice to unlearn first as that will make your journey much more enjoyable.

Relink to best solution: Here. Push side effects to the outer layer of you app; always.

4 Likes

When I was picking out a Halloween costume I wanted something that was scary to programmers, but I couldn’t figure out how to make global mutable state into a costume :person_shrugging: :smirk:

I’d encourage trying to set aside your OO experience (I’ve been there). And, yes, those ideas have been explored in Elixir, way back in 2016 here (I highly recommend the demo video).

Joking aside, yes you have discovered a real challenge: temporal coupling. I’m not sure you realize, but the ets solution will mean your tests can no longer run asynchronously, which would be a deal-breaker for me personally. It’s a great thing that the BEAM does not adhere to academic functional purity and we have options to manage shared state, such as ets (which Application.env is built upon), process dictionary, persistent terms, atomics, and counters.

A mocked module that looks in the process dictionary of the active test process for some offset would be my preference, but even that won’t work for tests that message other processes that fetch system time. In that case I would design the system with some processes that receive timestamps, but other “manager” code (that executes in the test process) that looks it up and passes it along. Yes, it requires discipline.

7 Likes