blubparadox

blubparadox

My elixir implementation of the 1 billion row challenge

Hi all. If you’ve looked on Twitter or YouTube you’ve seen the 1 billion row challenge, usually done in Java. I’ve written an Elixir version, feel free to have a look and see if you can improve on it.
https://github.com/rrcook/brc

Most Liked

garazdawi

garazdawi

Erlang Core Team

I built an Erlang implementation that does 1B lines in 60 seconds on my slow laptop.

https://github.com/jesperes/erlang_1brc/pull/1

To improve it further, maybe one could use the counters module. So much to do, so little time…

stevensonmt

stevensonmt

That post probably offers better insight than I can.

There’s also this blog post which offers the following explanation:

An ETS based approach
Other times, if the Agent doesn’t cut it for you, you might something faster. In these cases ETS might be a good option. The good thing about ETS is that it will always be faster because it doesn’t go through the Erlang Scheduler, furthermore it also supports concurrent reads and writes, which the Agent does not. However, it’s a bit more limited when you want to do atomic operations. Overall it’s very well suited for a simple shared key/value store, but if it’s better suited or not for your specific problem, that’s up to you.

Finally, the approach that immediately came to my mind was not using a cache at all but using Task.asyn_stream as described in this blog post
https://github.com/exercism/blog/blob/main/posts%2Fconcurrency-parallelism-in-elixir.md

tj0

tj0

Another implementation outside of the standard library if you’re interested.

Where Next?

Popular in RFCs Top

bartblast
:test_tube: Could use a hand testing something. Just landed a protocol pruning enhancement - the compiler now drops protocol implementati...
New
brettbeatty
At my work we build a lot of mix tasks for backfills and other maintenance tasks. Even with OptionParser we seem to copy a lot of boilerp...
New
simulacre7
Hi everyone, I’m building Aethrion, an early alpha Elixir runtime for persistent AI characters. Repo: GitHub - simulacre7/aethrion: A s...
New
KristerV
I got fed up with removing unused aliases manually so made a package for it: GitHub - KristerV/remove_unused_ex: Remove unused aliases an...
New
WyvernoDragon
I’ve been working on pgRx: a query engine that uses BEAM lightweight processes instead of PostgreSQL’s fork-per-connection model. The nu...
New
ca1989
Hi all, this week I played a bit with Phoenix and Elm. I ended up with this POC: https://github.com/carlotm/elmex The repository cont...
New
wingyplus
Hi, I start working on the fork version of Elixir GRPC. What I’ve done is the past few days: Support Elixir 1.11+ Support OTP 23+ OTP...
New
bortzmeyer
Not a replacement for serious authoritative DNS servers, of course, but useful for some uses (such as returning the IP address of the DNS...
New
KristerV
How I currently use Hexdocs I use hexdocs all day every day, but finding the right module and function takes too long even with my bookma...
New
pknoth
Built on top of boruta | Hex, I am on the way to creating a standalone OAuth 2.0/OpenID Connect server thinking of a lightweight Keycloak...
New

Other popular topics Top

albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 52341 488
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

We're in Beta

About us Mission Statement