technusm1

technusm1

Question regarding improving fizzbuzz IO performance

Hi everyone, posting for the first time here, so please bear with me.

I came across a HN thread that mentioned about a code-golf problem for high-throughput FizzBuzz. Link here: fastest code - High throughput Fizz Buzz - Code Golf Stack Exchange.

The naive single-threaded implementation in C gets about 70 MiB/s throughput on my Mac machine.
Here’s the C program for reference:

#include <stdio.h>

int main() {
    for (int i = 1; i < 1000000000; i++) {
        if ((i % 3 == 0) && (i % 5 == 0)) {
            printf("FizzBuzz\n");
        } else if (i % 3 == 0) {
            printf("Fizz\n");
        } else if (i % 5 == 0) {
            printf("Buzz\n");
        } else {
            printf("%d\n", i);
        }
    }
}

It is my understanding from learning Elixir so far that if your work is CPU intensive, pure Elixir is not a good choice. But since this is a simple problem (and mostly IO oriented as I’ve come to believe), I thought of testing the waters a bit. I came up with the approach of dividing the range 1..1000000000 into #{System.schedulers_online} chunks and using Task.async_stream to process each chunk in its own task.

I made the following Fizzbuzz module:

defmodule Fizzbuzz do
  def fizzbuzz_with_io(enumerable) do
    # I am printing here, because its my understanding it takes time to move data from one process to another.
    Stream.map(enumerable, &reply/1)
    |> Stream.chunk_every(5000)
    |> Enum.into(IO.stream())
  end

  def fizzbuzz_no_io(enumerable) do
    Stream.map(enumerable, &reply/1) |> Stream.run()
  end

  def reply(n) do
    case {rem(n, 3), rem(n, 5)} do
      {0, 0} -> "FizzBuzz\n"
      {0, _} -> "Fizz\n"
      {_, 0} -> "Buzz\n"
      {_, _} -> "#{n}\n"
    end
  end
end

My main CLI driver code (I acknowledge that the below code is not 100% correct and there are some numbers I might be missing in my quest to divide work, but it is OK enough to roughly illustrate the problem):

defmodule Fizzbuzz.Cli do
  def main([lower, upper]) do
    {lower, upper} = {String.to_integer(lower), String.to_integer(upper)}
    chunk_size = div(upper - lower, System.schedulers_online)
    input_enumerable = get_input_ranges(lower, upper, chunk_size)
    IO.inspect(input_enumerable)
    input_enumerable
    |> Task.async_stream(Fizzbuzz, :fizzbuzz, [], timeout: :infinity, max_concurrency: 64)
    |> Stream.run()
  end

  def main(_), do: IO.puts("Usage: fizzbuzz 1 10000")

  defp get_input_ranges(lower, upper, chunk_size) do
    if chunk_size >= 10 do
      if lower >= upper, do: [], else: [lower..min(lower+chunk_size, upper) | get_input_ranges(min(lower+chunk_size, upper) + 1, upper, chunk_size)]
    else
      [lower..upper]
    end
  end
end

Compiling above with MIX_ENV=prod mix escript.build, I have observed the following:

  • If I merely want to do the computation (with no IO at all using :fizzbuzz_no_io), the program finishes in 31 seconds on my machine. Its much faster than 3 minutes and 5 seconds of single threaded execution in Elixir, but its still slower than single-threaded C’s 7 seconds.
  • If I use :fizzbuzz_with_io, the program running time exceeds more than 5 minutes and I get a throughput of 2.5-2.7 MiB/s. Here, C gives a throughput of 70 MiB/s on my machine and a total running time of 1 minute and 43 seconds.

UPDATE-1:

  • I have updated the driver code to handle all cases properly.
  • I have updated fizzbuzz_with_io function to use Stream chunking, which has improved throughput to around 70 MiB/s on my system. That’s 35x improvement in performance. It was just a hunch that drove me to use this, since I thought adding chunking might add small chunks to IO.stream incrementally, rather than overloading it with everything at once. Could somebody please explain it better?
  • There are throughputs that still go in order of _ GiB/s. So, is there any further scope for improvement?

So, here’s my question: How do I improve the throughput of the Elixir version of Fizzbuzz program?
Thanks :smiley:

#io

Most Liked

ityonemo

ityonemo

Yep! One feature of the BEAM is that you can create a node, let’s say on your laptop, and connect into another node, let’s say in prod, using Erlang distribution and use the connection to debug what’s going on the remote node. Then let’s say you execute some code remotely that prints something to “stdio”. It’s useless if that IO gets sent to stdout on the remote node, if you’re lucky it gets logged but more likely it goes straight to /dev/null.

So Erlang will know that the code is being run remotely, assign that code’s group leader to your group leader on your laptop, and so IO calls to “stdout” in functions on the remote node will be forwarded to stdout on your laptop. To a first approximation, anyways. If you trigger code in a persistent genservers, that won’t hold true, for example, because that process holds on to its original group leader.

al2o3cr

al2o3cr

Two related gotchas with IO.stream:

  • it’s represented by a “file server” process, so “chunking” into bigger pieces helps by reducing the number of messages sent to that process (chunks vs individual lines)

  • it can still only print one message at a time, no matter how many workers are sending messages

You could likely boost performance by building bigger iolists and sending those to IO in one go.

mpope

mpope

Does IO.binwrite send a message to a process? Using raw file access means that no process communication happens, which could be saving time. maybe try this raw stdout access without flattening the binary, because iirc when you set the state in handle_info that isn’t a message pass to itself. I think state transitions are optimized.

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
JulienCorb
I am trying to implement my new.html.eex file to create new posts on my website. new.html.eex: &lt;h1&gt;Create Post&lt;/h1&gt; &lt;%= ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: The documentation above suggests that while ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29377 241
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43622 214
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36128 110
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

We're in Beta

About us Mission Statement