the_wildgoose

the_wildgoose

Elixir-reed_solomon_simd - wrappers for reed solomon encoding/ECC/FEC

Hi, I had a need for applying Reed Solomon protection to a (short) string. Think that you have a few hundred bytes and want to protect it against a couple of characters getting garbled, so you run it through the encoder, it creates a slightly longer string. Then later you can run (whatever makes it to the far end) through the decoder and repair any corruption. Think perhaps QR code protection, or protecting a long license key so it still works if you mistype, etc

However, a more common use case for Reed Solomon is to apply it to larger packets. For example PAR2 encoding slices a file into chunks and then encodes a protection stripped across all the chunks, so if a few were “erased”, you can recover them using the recovery packets

In a nutshell the encoding can be applied to even sized chunks and will protect against N/2 of those chunks going missing, where N is the amount of parity blocks you add

I wrapped two rust libraries. Why? Because I wrapped the high speed “reed-solomon-simd” library before realising that it wasn’t useful for my use case (protecting short strings), so I then wrapped a the simple reed-solomon-rs library which solves my use case. However, I’ve wrapped and released both libraries here:

https://github.com/ewildgoose/elixir-reed_solomon_simd

https://github.com/ewildgoose/rs_protect

Broadly you want the top one for protecting big lumps of data, such as IP packets, files, RAID on your disk, etc. And you want the second for encoding data on QR codes, or protecting small strings. Credit to the original authors as all I’m going is wrapping their work.

It’s my first attempt at writing some Rustler code, so feedback appreciated, also if anyone wants to give some guidance on how to pre-compile binaries for different platforms?

Most Liked

dimitarvp

dimitarvp

As I start I would recommend you not using .map_err to return a String. I’d make my own Error enum and have it implement Display (and maybe Debug as well) for such cases.

Or, even better, just implement rustler’s Encoder for it and map it to error tuples.

(Still use .map_err though, that’s how it’s done when you want to use ? and return Result-s but need your own custom error and want it serialized as an Elixir value.)

Or all of the above. I am doing that in my library (dimitarvp/xqlite on GitHub). Could send you a small PR if I have the time, though that’s not very likely, admittedly.

Where Next?

Popular in Announcing Top

danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29603 241
New
OvermindDL1
I created a new library (rather I pulled out a couple files from my big project), it manages an operating system PID file for the BEAM. ...
New
sasajuric
I’d like to announce a small library called boundaries. This is an experimental project which explores the idea of enforcing boundaries ...
New
josevalim
EDIT: since Ecto 3.0 final version is out, this post was amended to use the final versions in the instructions below. Hi everyone, We a...
New
tmbb
I’ve been working on two packages (not on hex.pm yet) to build admin interfaces for phoenix apps: bureaucrat - which contains a bunch ...
New
bryanjos
Hi, I just published version 0.23.0 of Elixirscript. https://github.com/bryanjos/elixirscript/blob/master/CHANGELOG.md Most of the chan...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
kip
Image is an image processing library for Elixir. It is based upon the fabulous vix library that provides a libvips wrapper for Elixir. I...
622 18934 194
New
aditya7iyengar
Rummage.Ecto and Rummage.Phoenix provide ways to perform Searching, Sorting and Pagination over Ecto queries and Phoenix collections. Fo...
New
fuelen
Hey folks! Want to present a toolkit for writing command-line user interfaces. It provides a convenient interface for colorizing text...
New

Other popular topics Top

sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 43487 311
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
saif
Hello everyone, Long time lurker first time poster here. I’ve recently begun working on Elixir full-time again! :raised_hands: It’s been...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 127089 1222
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New

We're in Beta

About us Mission Statement