Comparing memory consumption of programs run on BEAM

Hello,

I have a question concerning the comparison of memory consumption of programs run on BEAM.

Suppose that I have two versions of a program performing some computation, and I want to know which is consuming less memory. I want to have an automated testing system that runs two versions of the program multiple times in some isolated environment, measures their memory consumption during the run, and tells me which statistically uses less memory.

How should I perform the memory measurements? In particular:

1. What method should I use to fetch the current memory usage? Would :erlang.memory(:total) be enough, or should I subtract some constant “initial” value (e.g., the amount of memory used by the BEAM before running the program)? Or perhaps it would be better, to sum up the memory used by the particular processes building my program?
2. How often should I perform the measurement?
3. How should I deal with the randomness of GC invocations? The results of the tests seem to be susceptible to the moment when the GC is run.
4. How could a final metric (based on the memory measurements) look to give me a view on which version is, in fact, “better”?

I know that my question is really general, and answering it precisely requires some more specific definition of what it means that a particular program uses less memory than the other (do we take under consideration the maximum peak memory usage, or an average usage of memory, which type of memory are we interested in measuring, etc.), but I would appreciate some general tips on how such tests should be performed.

Best wishes,
Łukasz