When we say Elixir may not be suited to do number crunching, we are usually thinking a bit beyond analytics, averages, medians, etc. Because it is not necessarily the mathematical processing that hurts but rather the lack of support for large mutable data structures that would allow us to do implement things like image processing or support vector machines efficiently.
For example, think how you would implement a 100x100 matrix in Elixir and how much copying you would need to do to change a single {x,y} pair using immutable data structures. If you need performance, your best bet is ETS or falling back to C (which is what many languages do anyway).
If you are worried about analytics on something like Mint, Elixir/Phoenix should be fine.
