Function order changing for different environments

I’m finding a weird situation where I’m getting test failures in one environment and not another.

The following code handles my match correctly in CI, in my editor (spacemacs via Alchemist) and in a colleague’s terminal. But it consistently fails in my terminal (using the same elixir version). The code works correctly in production.

  defp app_platform(nil), do: :other # nil headers
  defp app_platform([]), do: :other  # no headers
  defp app_platform([<<"AppName"::utf8, "/", _version::binary>> | platform]) do # our typical mobile headers
    ...
  end

If I swap the 2nd and 3rd definitions the failures/successes swap around.

Note:

  • I use asdf for version management (my spacemacs config points to the mix shim).
  • Erlang/OTP 21 [erts-10.0.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]
  • Elixir 1.9.0 (compiled with Erlang/OTP 21)

In the failing and succeeding tests what are you passing into the app_platform function?

The user-agents I’m testing are:

  • “AppName/0.1.0 (com.redbubble; build:1; Android 28)”
  • “AppName/3.7.2 (com.redbubble; build:250; iOS Alamofire 53)”
  • “Mozilla/5.0 (Windows; rv:17.0) Gecko/20100101 Firefox/17.0”

We split the user agent on spaces and then try to use this to determine which platform it is:

  defp app_platform([<<"AppName"::utf8, "/", _version::binary>> | platform]) do
    cond do
      platform
      |> Enum.join(" ")
      |> String.match?(~r/android/i) ->
        :android

      platform
      |> Enum.join(" ")
      |> String.match?(~r/ios/i) ->
        :ios

      true ->
        :other
    end
  end

So, in the above, platform ends up being everything in the string after the /.
e.g. ["(com.redbubble;", "build:1;", "Android", "28)"]

AppName isn’t really the app name, but you get the gist.

Thanks @axelson, your question made me re-think my approach.
I ended up with this:

  defp app_platform(nil), do: :other
  defp app_platform([]), do: :other
  defp app_platform(["Android" | _]), do: :android
  defp app_platform(["iOS" | _]), do: :ios
  defp app_platform([_| platform]), do: app_platform(platform)

Still not sure why my original version worked in some places and not others, but this is working everywhere.

The code from your original post looks fine. What was the failing test checking? There shouldn’t be any difference in function clause matching between Elixir versions or runtime environments.

1 Like

Looks like I’ve worked out what’s going on here (if not why).

test

defmodule Mana.AppVersionTest do
  use ExUnit.Case
  alias Mana.AppVersion

  setup do
    {:ok,
     %{
       android: "AppName/0.1.0 (com.redbubble; build:1; Android 28)",
       ios: "AppName/3.7.2 (com.redbubble; build:250; iOS Alamofire 53)",
       other: "Mozilla/5.0 (Windows; rv:17.0) Gecko/20100101 Firefox/17.0"
     }}
  end

  describe "from_headers/1" do
    test "with android user agent", %{android: user_agent} do
      headers = ["User-Agent": user_agent]
      assert AppVersion.from_headers(headers) == {:android, "0.1.0"}
    end

    test "with ios user agent", %{ios: user_agent} do
      headers = ["User-Agent": user_agent]
      assert AppVersion.from_headers(headers) == {:ios, "3.7.2"}
    end

    test "with browser user agent", %{other: user_agent} do
      headers = ["User-Agent": user_agent]
      assert AppVersion.from_headers(headers) == {:other, nil}
    end
  end

  describe "app versions" do
    test "Grab the version from the user agent for iOS", %{ios: user_agent} do
      headers = ["User-Agent": user_agent]
      assert AppVersion.app_version_from_headers(headers) == "3.7.2"
    end

    test "Grab the version from the user agent for Android", %{android: user_agent} do
      headers = ["User-Agent": user_agent]
      assert AppVersion.app_version_from_headers(headers) == "0.1.0"
    end

    test "Grab the version from the user agent if it's just the version number" do
      headers = ["User-Agent": "3.8.0"]
      assert AppVersion.app_version_from_headers(headers) == "3.8.0"
    end

    test "Returns nil if it doesn't find the version" do
      headers = ["User-Agent": "AppName"]
      assert AppVersion.app_version_from_headers(headers) == nil
    end

    test "Returns nil if it is passed in nil" do
      headers = ["User-Agent": nil]
      assert AppVersion.app_version_from_headers(headers) == nil
    end
  end

  describe "app platform" do
    test "for iOS", %{ios: user_agent} do
      headers = ["User-Agent": user_agent]
      assert AppVersion.app_platform_from_headers(headers) == :ios
    end

    test "for Android", %{android: user_agent} do
      headers = ["User-Agent": user_agent]
      assert AppVersion.app_platform_from_headers(headers) == :android
    end

    test "when headers are empty" do
      assert AppVersion.app_platform_from_headers([]) == :other
    end

    test "when not specified", %{other: user_agent} do
      headers = ["User-Agent": user_agent]
      assert AppVersion.app_platform_from_headers(headers) == :other
    end
  end
end

implementation

defmodule Mana.AppVersion do
  @moduledoc """
  Extract the app version and platform from the user agent.
  """

  def from_headers(headers) do
    case headers |> Keyword.fetch(:"User-Agent") do
      :error ->
        nil

      {:ok, _user_agent} ->
        {
          headers |> app_platform_from_headers(),
          headers |> app_version_from_headers()
        }
    end
  end

  def app_version_from_headers(nil), do: nil

  def app_version_from_headers(headers) do
    case headers |> Keyword.fetch(:"User-Agent") do
      :error ->
        nil

      {:ok, user_agent} ->
        user_agent
        |> app_version()
    end
  end

  def app_platform_from_headers(nil), do: nil

  def app_platform_from_headers(headers) do
    case headers |> Keyword.fetch(:"User-Agent") do
      :error ->
        :other

      {:ok, user_agent} ->
        user_agent
        |> String.split(" ")
        |> app_platform()
    end
  end

  # App version detection

  defp app_version(nil), do: nil

  defp app_version(user_agent) do
    result =
      user_agent
      |> String.split(" ")
      |> List.first()
      |> String.split("/")
      |> List.last()

    case Version.parse(result) do
      {:ok, _} -> result
      _ -> nil
    end
  end

  # App platform detection

  defp app_platform(nil), do: :other
  defp app_platform([]), do: :other

  defp app_platform([<<"AppName"::utf8, "/", _version::binary>> | platform]) do
    cond do
      platform
      |> Enum.join(" ")
      |> String.match?(~r/android/i) ->
        :android

      platform
      |> Enum.join(" ")
      |> String.match?(~r/ios/i) ->
        :ios

      true ->
        :other
    end
  end

  defp app_platform([_ | _]), do: :other
end

error output

  1) test app platform for iOS (Mana.AppVersionTest)
     test/mana/app_version_test.exs:59
     Assertion with == failed
     code:  assert AppVersion.app_platform_from_headers(headers) == :ios
     left:  :other
     right: :ios
     stacktrace:
       test/mana/app_version_test.exs:61: (test)
....
  2) test app platform for Android (Mana.AppVersionTest)
     test/mana/app_version_test.exs:64
     Assertion with == failed
     code:  assert AppVersion.app_platform_from_headers(headers) == :android
     left:  :other
     right: :android
     stacktrace:
       test/mana/app_version_test.exs:66: (test)
.
  3) test from_headers/1 with ios user agent (Mana.AppVersionTest)
     test/mana/app_version_test.exs:20
     Assertion with == failed
     code:  assert AppVersion.from_headers(headers) == {:ios, "3.7.2"}
     left:  {:other, "3.7.2"}
     right: {:ios, "3.7.2"}
     stacktrace:
       test/mana/app_version_test.exs:22: (test)
.
  4) test from_headers/1 with android user agent (Mana.AppVersionTest)
     test/mana/app_version_test.exs:15
     Assertion with == failed
     code:  assert AppVersion.from_headers(headers) == {:android, "0.1.0"}
     left:  {:other, "0.1.0"}
     right: {:android, "0.1.0"}
     stacktrace:
       test/mana/app_version_test.exs:17: (test)

Finished in 0.07 seconds
12 tests, 4 failures

success case

  • modify the implementation in my editor (spacemacs)
  • run the test in my editor
  • test passes

failure case

  • modify the implementation in my editor
  • run the test in my terminal
  • test fails

The failure case can be reproduced in either direction. If you edit the implementation, wherever you run the test next will pass. If you then run the test in the opposite environment (without requiring recompilation) the test will fail. So, it seems to me there’s a difference in the compilation between the two envs.

I have this in my .spacemacs config:

   alchemist-mix-command "~/.asdf/shims/mix"
   alchemist-execute-command "~/.asdf/shims/elixir"
   alchemist-compile-command "~/.asdf/shims/elixirc"

The paths to the executables:

craigread@C02T17ZGGTFM /Users/craigread/src/mana/app                                              2.2.8 18a1ba5
⚡ which mix                                                                         (production-us-east-1/default)
/Users/craigread/.asdf/shims/mix

craigread@C02T17ZGGTFM /Users/craigread/src/mana/app                                              2.2.8 18a1ba5
⚡ which elixirc                                                                     (production-us-east-1/default)
/Users/craigread/.asdf/shims/elixirc

craigread@C02T17ZGGTFM /Users/craigread/src/mana/app                                              2.2.8 18a1ba5
⚡ which elixir                                                                      (production-us-east-1/default)
/Users/craigread/.asdf/shims/elixir

The recursive version of this code doesn’t have the same behaviour.

Have you tried running property tests to check if this isn’t caused by some implementation problems and to find minimal set of values that fail?

No, I haven’t. That’s a good idea. Thanks!

edit

property test

    test "with random android user agent" do
      check all major <- StreamData.map(StreamData.positive_integer(), &Integer.to_string/1),
                minor <- StreamData.map(StreamData.positive_integer(), &Integer.to_string/1),
                patch <- StreamData.map(StreamData.positive_integer(), &Integer.to_string/1),
                build <- StreamData.map(StreamData.positive_integer(), &Integer.to_string/1),
                other_build <- StreamData.map(StreamData.positive_integer(), &Integer.to_string/1) do
        version = major <> "." <> minor <> "." <> patch
        user_agent = "AppName/" <> version <> " (com.redbubble; build:" <> build <> "; Android " <> other_build <> ")"

        headers = ["User-Agent": user_agent]
        assert AppVersion.from_headers(headers) == {:android, version}
      end
    end

First time I’ve tried doing any property testing, so not sure if the above is the greatest test.
However, it’s still displaying the same behaviour on the simplest example.

test output

  1) test from_headers/1 with random android user agent (Mana.AppVersionTest)
     test/mana/app_version_test.exs:17
     Failed with generated values (after 0 successful runs):

         * Clause:    major <- StreamData.map(StreamData.positive_integer(), &Integer.to_string/1)
           Generated: "1"

         * Clause:    minor <- StreamData.map(StreamData.positive_integer(), &Integer.to_string/1)
           Generated: "1"

         * Clause:    patch <- StreamData.map(StreamData.positive_integer(), &Integer.to_string/1)
           Generated: "1"

         * Clause:    build <- StreamData.map(StreamData.positive_integer(), &Integer.to_string/1)
           Generated: "1"

         * Clause:    other_build <- StreamData.map(StreamData.positive_integer(), &Integer.to_string/1)
           Generated: "1"

     Assertion with == failed
     code:  assert AppVersion.from_headers(headers) == {:android, version}
     left:  {:other, "1.1.1"}
     right: {:android, "1.1.1"}
     stacktrace:
       test/mana/app_version_test.exs:27: anonymous fn/6 in Mana.AppVersionTest."test from_headers/1 with random android user agent"/1
       (stream_data) lib/stream_data.ex:2072: StreamData.check_all/7
       test/mana/app_version_test.exs:18: (test)

Same issue as before: If compilation was triggered in the place where the test was run, the test passed. Otherwise, it failed.

Hmm, I just printed out the value of platform in the app_platform function. That is getting called every time (pass or fail), so it must be the regex that’s failing. But it succeeds in iex.

iex(12)> ["(com.redbubble;", "build:1;", "Android", "1)"] |> Enum.join(" ") |> String.match?(~r/android/i)
true

I looked at your code and it looks it should have worked. I copied the test and the offending module and ran the tests locally and it all passed.

I don’t see how it could behave differently when compiling the code from Spacemacs and compiling the code from the terminal. It looks like both are pointing to the same versions of the elixir compiler.

Are you able to consistently reproduce this issue?

Yes, the issue is totally reproducible on my machine.
It’s not really a problem for me, just an intriguing mystery.

I was sure it had to be environmental, so I (just now) tested what happens when switching from one terminal to another. If I run the tests the first time in iTerm, kitty or terminal (causing the code to be compiled there), they run fine in another terminal (but not in my editor).
And the issue is also present when switching from tests run/compiled via alchemist and switching to eterm or multi-term inside my editor.

That makes me think it must be alchemist (or spacemacs config) related, but I don’t know how. I might need to test my spacemacs config on a colleague’s machine next week. My config is here if anybody else is interested enough to test that theory.

1 Like