Elixir-reed_solomon_simd - wrappers for reed solomon encoding/ECC/FEC

Your Libraries & OS Mentoring Libraries
, , , ,
1

Hi, I had a need for applying Reed Solomon protection to a (short) string. Think that you have a few hundred bytes and want to protect it against a couple of characters getting garbled, so you run it through the encoder, it creates a slightly longer string. Then later you can run (whatever makes it to the far end) through the decoder and repair any corruption. Think perhaps QR code protection, or protecting a long license key so it still works if you mistype, etc

However, a more common use case for Reed Solomon is to apply it to larger packets. For example PAR2 encoding slices a file into chunks and then encodes a protection stripped across all the chunks, so if a few were “erased”, you can recover them using the recovery packets

In a nutshell the encoding can be applied to even sized chunks and will protect against N/2 of those chunks going missing, where N is the amount of parity blocks you add

I wrapped two rust libraries. Why? Because I wrapped the high speed “reed-solomon-simd” library before realising that it wasn’t useful for my use case (protecting short strings), so I then wrapped a the simple reed-solomon-rs library which solves my use case. However, I’ve wrapped and released both libraries here:

Broadly you want the top one for protecting big lumps of data, such as IP packets, files, RAID on your disk, etc. And you want the second for encoding data on QR codes, or protecting small strings. Credit to the original authors as all I’m going is wrapping their work.

It’s my first attempt at writing some Rustler code, so feedback appreciated, also if anyone wants to give some guidance on how to pre-compile binaries for different platforms?

2 Likes
2

As I start I would recommend you not using .map_err to return a String. I’d make my own Error enum and have it implement Display (and maybe Debug as well) for such cases.

Or, even better, just implement rustler’s Encoder for it and map it to error tuples.

(Still use .map_err though, that’s how it’s done when you want to use ? and return Result-s but need your own custom error and want it serialized as an Elixir value.)

Or all of the above. I am doing that in my library (dimitarvp/xqlite on GitHub). Could send you a small PR if I have the time, though that’s not very likely, admittedly.

1 Like
3

That’s a very helpful reply! Thank you!

I concede I’ve little idea what I’m going here… I am cargo culting a lot, and struggled significantly to figure out how to get the rustler mapping to work. I’ve zero idea where to start on those suggestions, but I’ll take a peek at your project, thanks for the link!

Obviously will not turn down PRs! Thanks

4

Find the error.rs file in my project, and then look for impl rustler::Error for XqliteError (or just impl Error).

That should unblock you IMO.