Erlang and Elixir on Apple Silicon/M1 Chip

does this help: Anyone received their new MacBook Pros today? - #8 by joseph-lozano

Unfortunately it has not helped.

Using asdf to uninstall and then reinstall Erlang with the following flags exported definitely made a big difference on my MacBook Pro M1 Max.

export KERL_CONFIGURE_OPTIONS="
      --disable-debug \
      --disable-silent-rules \
      --enable-dynamic-ssl-lib \
      --enable-hipe \
      --enable-shared-zlib \
      --enable-smp-support \
      --enable-threads \
      --enable-wx \
      --with-ssl=$(brew --prefix openssl@1.1) \
      --without-javac
      --enable-darwin-64bit \
      --enable-kernel-poll \
      --with-dynamic-trace=dtrace \
     "
1 Like

Enable hipe? Isnt that project dead?

1 Like

The args were copied from the Erlang homebrew formula.

Given this one is maintained by brew bump-formula-pr is likely keeping cruft around quite well :smiley:

1 Like

I’m running almost those exact options, I have --disable-hipe instead of enable since it’s a dead project.

So here’s something that’s interesting. I did rm -rf _build and then iex -S mix run --no-start --no-compile so that I could open :observer, and then I called the Mix.Task.run("deps.compile") to look at how it performed from Erlang’s perspective.

Here’s what I saw:

Initial:

More deps:

More deps:

This looks super strange to me. Scheduler utilization across the board is super low. I ran compile after the deps were compiled and although it did look like all 10 schedulers got involved (as we see here with some deps) none of them ever exceeded 30%.

I’m not entirely sure what to make of this. The big memory spike was Poison (yes, I know) and I think that’s likely just code gen. Anyone got any insights?

UPDATE:

Running this code trivially uses all 10 schedulers and maxes out cores from an OS level:

iex(3)> defmodule Busy do
  ...(3)>   def loop(x) do
  ...(3)>     loop(x + 1)
  ...(3)>   end
  ...(3)> end
  iex(4)> 1..20 |> Enum.map(fn _ -> spawn(fn -> Busy.loop(0) end)

So maybe the OS is right. Compilation (for some reason) is generally not generating very much scheduler activity (for some reason) and so it can run everything on the efficiency cores. When the schedulers are under load it does seem to happily move them to the performance cores.

5 Likes

IO bound then?

Very unlikely that compilation is that IO bound. He is using about 5% of the processing power. Maybe if sth is wrong with the storage.

After a chat with @dch on IRC we’re noticing that :erlang.system_info :cpu_topology returns :undefined which may be part of the issue. Will update with followup.

7 Likes

Interesting think is that on my MacBook with Intel it also returns :undefined.

The cpu_topology only affects how schedulers are bound. So you have to pass the +sbt option to Erlang in order for it to change anything. If I recall correctly it will return undefined on all MacOS devices because Apple does not provide (or we at least do not fetch) information about what the CPU topology looks like.

3 Likes

It seems that OSX (on both intel, which I have here, and M1, which I don’t) doesn’t provide any topology information, so we get undefined everywhere. For comparison, an 8 core non-SMT intel box (single socket):

 erl -noshell -eval 'io:format("~p~n",[erlang:system_info(cpu_topology)])' -s erlang halt
[{processor,[{core,{logical,0}},
             {core,{logical,1}},
             {core,{logical,2}},
             {core,{logical,3}},
             {core,{logical,4}},
             {core,{logical,5}},
             {core,{logical,6}},
             {core,{logical,7}}]}]

And an 80-core arm64 box with partitioning (single socket, but NUMA):

[{processor,[{core,{logical,0}},
             {core,{logical,1}},
             {core,{logical,2}},
              ...
             {core,{logical,39}}]},
 {processor,[{core,{logical,40}},
             {core,{logical,41}},
             {core,{logical,42}},
             ...
             {core,{logical,79}}]}]

It would be great to see if the OTP team could advise how to (ab)use the cores best here on M1, given that the efficiency cores run ~ 1/3 slower clock speed. It would be great to pin the schedulers, but I guess that requires Apple to provide information on how to access it.

The usual FreeBSD sysctls that expose that seem to be missing on OSX.

1 Like

More M1 fun: Hardware Information - by Jim Cownie - CPU fun is the best resource I found… in particular brew install hwloc and running lstopo. neat!

Setting up the Apple M1 for Code Development is also excellent.

2 Likes

[hwloc-announce] hwloc 2.6.0 released it seems the latest release is able to extract CPU info from M1


Version 2.6.0
-------------
* Backends
  + Expose two cpukinds for energy-efficient cores (icestorm) and
    high-performance cores (firestorm) on Apple M1 on Mac OS X.

Maybe that helps!

1 Like

Linking this back to OTP itself, configure looks for a number of platform-specific cpu pinning options, and sets HAVE_CPUSET_xETAFFINITY during configure using some configure.ac checks. This requires the sys.cpuset.h headers.

On OSX, this simply isn’t here. Skimming Apple’s docs, I the closest I could find is here Affinity API Release Notes for OS X v10.5 which is pretty old now, but suggests we don’t have that functionality at all, certainly at least back in Snow Leopard days.

I dug around on OSX and found /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/mach/thread_policy.h & found this gem, which appears to be as close as we can get.

/*
 * THREAD_AFFINITY_POLICY:
 *
 * This policy is experimental.
 * This may be used to express affinity relationships
 * between threads in the task. Threads with the same affinity tag will
 * be scheduled to share an L2 cache if possible. That is, affinity tags
 * are a hint to the scheduler for thread placement.
 *
 * The namespace of affinity tags is generally local to one task. However,
 * a child task created after the assignment of affinity tags by its parent
 * will share that namespace. In particular, a family of forked processes
 * may be created with a shared affinity namespace.
 *
 * Parameters:
 * tag: The affinity set identifier.
 */

On FreeBSD, which supports scheduler binding, the configure script shows:

checking for sched_getaffinity/sched_setaffinity... no
checking for pset functionality... no
checking for processor_bind functionality... no
checking for cpuset_getaffinity/cpuset_setaffinity... yes

vs @benwilson512 's OSX M1:

checking for sched_getaffinity/sched_setaffinity... no
checking for pset functionality... no
checking for processor_bind functionality... no
checking for cpuset_getaffinity/cpuset_setaffinity... no
2 Likes