technusm1
Question regarding improving fizzbuzz IO performance
Hi everyone, posting for the first time here, so please bear with me.
I came across a HN thread that mentioned about a code-golf problem for high-throughput FizzBuzz. Link here: fastest code - High throughput Fizz Buzz - Code Golf Stack Exchange.
The naive single-threaded implementation in C gets about 70 MiB/s throughput on my Mac machine.
Here’s the C program for reference:
#include <stdio.h>
int main() {
for (int i = 1; i < 1000000000; i++) {
if ((i % 3 == 0) && (i % 5 == 0)) {
printf("FizzBuzz\n");
} else if (i % 3 == 0) {
printf("Fizz\n");
} else if (i % 5 == 0) {
printf("Buzz\n");
} else {
printf("%d\n", i);
}
}
}
It is my understanding from learning Elixir so far that if your work is CPU intensive, pure Elixir is not a good choice. But since this is a simple problem (and mostly IO oriented as I’ve come to believe), I thought of testing the waters a bit. I came up with the approach of dividing the range 1..1000000000 into #{System.schedulers_online} chunks and using Task.async_stream to process each chunk in its own task.
I made the following Fizzbuzz module:
defmodule Fizzbuzz do
def fizzbuzz_with_io(enumerable) do
# I am printing here, because its my understanding it takes time to move data from one process to another.
Stream.map(enumerable, &reply/1)
|> Stream.chunk_every(5000)
|> Enum.into(IO.stream())
end
def fizzbuzz_no_io(enumerable) do
Stream.map(enumerable, &reply/1) |> Stream.run()
end
def reply(n) do
case {rem(n, 3), rem(n, 5)} do
{0, 0} -> "FizzBuzz\n"
{0, _} -> "Fizz\n"
{_, 0} -> "Buzz\n"
{_, _} -> "#{n}\n"
end
end
end
My main CLI driver code (I acknowledge that the below code is not 100% correct and there are some numbers I might be missing in my quest to divide work, but it is OK enough to roughly illustrate the problem):
defmodule Fizzbuzz.Cli do
def main([lower, upper]) do
{lower, upper} = {String.to_integer(lower), String.to_integer(upper)}
chunk_size = div(upper - lower, System.schedulers_online)
input_enumerable = get_input_ranges(lower, upper, chunk_size)
IO.inspect(input_enumerable)
input_enumerable
|> Task.async_stream(Fizzbuzz, :fizzbuzz, [], timeout: :infinity, max_concurrency: 64)
|> Stream.run()
end
def main(_), do: IO.puts("Usage: fizzbuzz 1 10000")
defp get_input_ranges(lower, upper, chunk_size) do
if chunk_size >= 10 do
if lower >= upper, do: [], else: [lower..min(lower+chunk_size, upper) | get_input_ranges(min(lower+chunk_size, upper) + 1, upper, chunk_size)]
else
[lower..upper]
end
end
end
Compiling above with MIX_ENV=prod mix escript.build, I have observed the following:
- If I merely want to do the computation (with no IO at all using
:fizzbuzz_no_io), the program finishes in 31 seconds on my machine. Its much faster than 3 minutes and 5 seconds of single threaded execution in Elixir, but its still slower than single-threaded C’s 7 seconds. - If I use
:fizzbuzz_with_io, the program running time exceeds more than 5 minutes and I get a throughput of 2.5-2.7 MiB/s. Here, C gives a throughput of 70 MiB/s on my machine and a total running time of 1 minute and 43 seconds.
UPDATE-1:
- I have updated the driver code to handle all cases properly.
- I have updated fizzbuzz_with_io function to use Stream chunking, which has improved throughput to around 70 MiB/s on my system. That’s 35x improvement in performance. It was just a hunch that drove me to use this, since I thought adding chunking might add small chunks to IO.stream incrementally, rather than overloading it with everything at once. Could somebody please explain it better?
- There are throughputs that still go in order of _ GiB/s. So, is there any further scope for improvement?
So, here’s my question: How do I improve the throughput of the Elixir version of Fizzbuzz program?
Thanks ![]()
Most Liked
ityonemo
Yep! One feature of the BEAM is that you can create a node, let’s say on your laptop, and connect into another node, let’s say in prod, using Erlang distribution and use the connection to debug what’s going on the remote node. Then let’s say you execute some code remotely that prints something to “stdio”. It’s useless if that IO gets sent to stdout on the remote node, if you’re lucky it gets logged but more likely it goes straight to /dev/null.
So Erlang will know that the code is being run remotely, assign that code’s group leader to your group leader on your laptop, and so IO calls to “stdout” in functions on the remote node will be forwarded to stdout on your laptop. To a first approximation, anyways. If you trigger code in a persistent genservers, that won’t hold true, for example, because that process holds on to its original group leader.
al2o3cr
Two related gotchas with IO.stream:
-
it’s represented by a “file server” process, so “chunking” into bigger pieces helps by reducing the number of messages sent to that process (chunks vs individual lines)
-
it can still only print one message at a time, no matter how many workers are sending messages
You could likely boost performance by building bigger iolists and sending those to IO in one go.
mpope
Does IO.binwrite send a message to a process? Using raw file access means that no process communication happens, which could be saving time. maybe try this raw stdout access without flattening the binary, because iirc when you set the state in handle_info that isn’t a message pass to itself. I think state transitions are optimized.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance








