I can confirm this issue. As I have managed to reproduce it @emeryotopalik

And it sounds like a serious one. And when it fails, it fails repeatedly for a while until it may go back to normal, and then it may fail again.

I am running Arch Linux 64bits, and my elixir --version is:
OTP 24.1.2

$ elixir --version
Erlang/OTP 24 [erts-12.1.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Elixir 1.12.2 (compiled with Erlang/OTP 24)

mix test fails big time:

$ mix test
Compiling 3 files (.ex)
Compiling lib/bar.ex (it's taking more than 10s)
Generated debug_me app

  1) test Compile time (FooTest)
     test/foo_test.exs:19
     Assertion with == failed
     code:  assert report(Bar.result()) == [false: times]
     left:  [{false, 1970374}, {true, 8029626}]
     right: [false: 10000000]
     stacktrace:
       test/foo_test.exs:22: (test)

.

  2) test Runtime (FooTest)
     test/foo_test.exs:12
     Assertion with == failed
     code:  assert report(result) == [false: times]
     left:  [
              {false, 606},
              {true, 640},
              {false, 588},
              {true, 1264},
              {false, 597},
              {true, 528},
              {false, 194331},
              {true, 2000},
              {false, 1333},
              {true, 1333},
              {false, 666},
              {true, 9796114}
            ]
     right: [false: 10000000]
     stacktrace:
       test/foo_test.exs:16: (test)
..

Finished in 5.1 seconds (0.00s async, 5.1s sync)
1 doctest, 4 tests, 2 failures

Randomized with seed 934406

$ MIX_ENV=test iex -S mix
Erlang/OTP 24 [erts-12.1.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Interactive Elixir (1.12.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> for chunk <- Enum.chunk_by(Bar.result(), & &1), do: {hd(chunk), length(chunk)}
[false: 1970374, true: 8029626]

iex(2)> Enum.map(1..10_000_000, fn _i -> Foo.eval(nil, 20) end) |> Foo.report()       
[false: 10000000]

iex(3)> Enum.map(1..10_000_000, fn _i -> Foo.eval(nil, 20) end) |> Foo.report()
[false: 10000000]

iex(4)> Enum.map(1..10_000_000, fn _i -> Foo.eval(nil, 20) end) |> Foo.report()
[false: 10000000]

Now I run it with OTP 23.3.4.9 , mix testpass all tests,
here are the commands on IEx:

$ iex -S mix
Erlang/OTP 23 [erts-11.2.2.8] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Interactive Elixir (1.12.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> for chunk <- Enum.chunk_by(Bar.result(), & &1), do: {hd(chunk), length(chunk)}
[false: 10000000]

iex(2)> Enum.map(1..10_000_000, fn _i -> Foo.eval(nil, 20) end) |> Foo.report()
[false: 10000000]

…and it’s all good.

So my quick assumption is that this is due the the JIT compiler introduced in OTP24

I have created a project for things like this, where I test an Elixir project in every Elixir/OTP version combination.
and suprisingly it does not fail for OTP 24 with JIT. (it only fails with OTP 17, and it does not even compile, so that is unrelated).

https://github.com/eksperimental/debug_me/runs/4491828158?check_suite_focus=true

the source code of Elixir project can be found here:
GitHub - eksperimental/debug_me at emeryotopalik

UPDATE 1:

And Noticed that when it fails it fails after ~600 results (or multiple of this number), after running MIX_ENV=test mix do compile --force, test multiple times:

these 2 are from the same compilation:

# Compile times:  1998
[{false, 1246}, {true, 1186}, {false, 666}, {true, 1126}, {false, 666}, {true, 5110}]

# Run time: Note that 1998 is 666 * 3
[{false, 1998}, {true, 8002}]
----------------

then I got:

# Runtime (but Compile time did not fail)
[{false, 606}, {true, 999394}]

--------
# Compiletime
 [{false, 665}, {true, 99335}]

---------
# Compile time
[{false, 665}, {true, 99335}]

Exceptions aren’t propagated outside the guard, they are considered “failure to match”:

If an arithmetic expression, a Boolean expression, a short-circuit expression, or a call to a guard BIF fails (because of invalid arguments), the entire guard fails. If the guard was part of a guard sequence, the next guard in the sequence (that is, the guard following the next semicolon) is evaluated.

(from the Erlang reference manual)

I’d guess the intent was to avoid having to spam is_integer and similar checks.

I tried your code. Foo.eval(nil, 20) always returns false to me.

iex(36)> 1..10000 |> Enum.map(fn _ -> Foo.eval(nil, 20) end) |> Enum.uniq
[false]