vans163

vans163

So Iv been working on Morphlings diff algo to create a dom differential to send over the wire wasting as little CPU/Memory as possible and wanted to talk about a point when dealing with binaries.

This is how you shoot yourself in the foot and then turn around and say BEAM/Erlang/Elixir is very slow and bad.

100x Slower code with linear slowdown

    fn_chunk = fn(bin,parts)->
        size = byte_size(bin)
        segment_size = div(size, parts)+1
        {chunks,_} = Enum.reduce_while(0..(parts*2), {[], bin}, fn(idx,{a,bin})->
            piece = String.slice(bin, 0, segment_size)
            case piece do
                "" -> {:halt, {a,bin}}
                _ ->
                    a = a ++ [%{old_pos: idx*segment_size, size: segment_size, binary: piece}]
                    {:cont, {a, String.slice(bin, segment_size, size)}}
            end
        end)
        chunks
    end

Fast code

fn_chunk = fn(bin,parts)->
        size = byte_size(bin)
        segment_size = div(size, parts)+1
        {chunks,_} = Enum.reduce_while(0..(parts*2), {[], 0}, fn(_,{a,idx})->
            to_take = min(byte_size(bin)-idx, segment_size)
            piece = :binary.part(bin, idx, to_take)
            case piece do
                "" -> {:halt, {a,bin}}
                _ ->
                    a = a ++ [%{old_pos: idx, size: to_take, binary: piece}]
                    {:cont, {a, idx+to_take}}
            end
        end)
        chunks
    end

What this piece does is, it splits a binary into x amount of equal parts. On a binary of size 20k, the above slow code takes 100ms, on a 5k size binary takes 43ms.

The fast code takes 0-1ms on a 20k size binary.

What is the difference?

Showing Posts 1 to 4

sneako

sneako

String.slice/3 works on Unicode graphemes vs :binary.part/3 which is just looking at raw bytes. The documentation mentions this and suggests Kernel.binary_part/3 if you dont care about graphemes String — Elixir v1.20.2

xlphs

xlphs

What if you just pattern match? << part :: binary-size(n), rest :: binary>> = bin where n is number of bytes for each part.

vans163

vans163 OP

There is a lot of options to consider, what I was getting at is its a little confusing to know which operations make copies and which refer to the original binary on the shared binary heap. And this was a good performance refresher for me.

ananthakumaran

ananthakumaran

I would assume the second version is using sub binary. Instead of allocating a new binary, a reference to the original is created. I agree, from the documentation it’s not clear which function will allocate a new one vs share the original binary.

Be careful though, the original binary will not be garbage collected till all the sub binaries are garbage collected. This has a tendency to cause memory leak. Let’s say you take the header part of a large binary and keep it in genserver state, the garbage collection of the large binary will be delayed. binary:copy/1 can be used to create a independent copy. NimbleCSV — NimbleCSV v1.3.0

— All posts loaded —

Where Next? Top

Trending in Discussions Top

AstonJ
As the title says, please share what you’ve been up to with Elixir. Whether that’s been learning it, looking into it, making stuff with i...
2977 94592 917
New
cblavier
Hey there, It’s been more than a year since we started using LiveView as our main UI library and building a whole library of UI componen...
New
mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
heathen
Quite interesting article Google brought me. Didn’t find any mentions about it here. What do you think in general? Would you use togethe...
New
mhanberg
Hi everyone! The first release candidate for the Expert language server project is now available! We’ve published a press release detai...
New
AstonJ
Since we have deprecated our Erlang sections (as we have dedicated Erlang Forums now) let’s add this thread for those who’d like to post ...
New
maennchen
:warning: Security advisory: Decimal DoS vulnerability A vulnerability has been published for decimal where very large exponents can cau...
New

Other Trending Topics Top

JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
marciok
Hi there! We created Gust: A task orchestrator inspired by Airflow. For those who have never heard about Aiflow, it’s a Python-based wor...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Dmk
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New
netoum
Corex is an accessible, unstyled UI component library for Phoenix that integrates Zag.js state machines using Vanilla JavaScript and Live...
New
webofbits
With AI doing more of the implementation work, I’ve been wondering how much coding I should deliberately keep doing myself. My main conc...
#ai
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews