HyperLogLog is one of my favorite probabilistic algorithms. It’s a super space-efficient way for cardinality estimation. I recently used it at work via Presto and Spark and it works surprisingly well. Then I decided to make it from scratch to have better understanding how it actually works and chose lovely Elixir to implement it.

The rationale to make it Redis compatible is mostly because it’s an easy way to ensure that I have the correct implementation by checking that given same inputs, Redis and this module would yield the same output.

The Redis-compatible module used the same hash function (port it to Elixir), same estimation algorithm and same binary format. Therefore, you can use it to interact with Redis (v5) as well.

The repo is available at https://github.com/gyson/hll

P.S. During development, I found that binary pattern matching in Elixir/Erlang is pretty sweet for binary encoding and decoding !