Publishing package with third party dependencies written in C (like boringssl)

Hi,
I am writing library that runs somoe code in C. This code uses some other code written in C e.g. boringssl. In my repository I have specified boringssl via git submodule and it’s okay for development phase. I can download it manualy with git submodule init .... Now I want to release this library so that user don’t have to install any C libs in his OS. I tried to publish source code of boringssl (and other third party C deps) with my lib but the size of packgae is too big (> 8 MB compressed). My question is how should I publish such package. Maybe I shouldn’t publish source files of boringssl and write own mix task to download needed files before compilation. But in this way user has to have installed git in his OS ( I need to clone somehow this repo using e.g. System.cmd()).

1 Like

I think it’s best not to put any clever scripts inside your library; distribute everything as-is.

Compiling C as part of a higher-level library is usually ridden with traps so I’d opt for just precompiling the C dependencies for all platforms you expect your library to be used on. But of course, that might inflate your library’s file size even further.

Finally, you can just use the Rust bindings and opt to use boringssl through rustler but that might be yet another can of worms. That path could be slightly less nasty (since the Rust library takes of care of having boringssl's C sources) but can’t say for sure before trying. I know that rusqlite (which is Rust bindings for sqlite3, another C library) offers a precompiled sqlite3 dynamic library if it can’t be compiled on the user’s system. Sadly boringssl's Rust bindings don’t seem to offer the same option.

Finally, there are several pure Rust libraries (without C libraries beneath) that implement a good part of boringssl but I haven’t made a thorough research beyond just knowing that.

Thanks for reply!

The problem is that I am not using boringssl directly. I use lsquic which is QUIC implementation in C and lsquic uses boringssl so to compile lsquic I have to compile boringssl at first so I can’t switch to any other implementation of SSL (or bindings).

Currently I am wrapping C code using Unifex and Bundlex so using Rustler is also not an option at this moment :confused:

I also agree that cloning repositories before compilation doesn’t sound too good. For me the best option will be to include boringssl source files and compile them before compiling my lib. In such scenario all platforms supported by boringssl will be also supported by my lib.
(quiche does it in such way quiche 0.7.0 - Docs.rs)

If you are curious the code is available here
It’s released but importing dependency in your project with {:ex_quic, "~> 0.1.2"} will not work at this moment.

I see. Sad that you can’t use Rust directly, that’s what I’d always recommend after having direct experience in writing Elixir libraries with Rust bindings beneath. It’s much less troublesome.

I used quiche once in a toy Rust project and loved it, was very seamless.

Can’t you precompile lsquic in Docker containers and then extract all precompiled versions inside your library? Could be the easier path; looking at the Git repo there aren’t many platforms that are supported by it. I count 7.

Sounds good but I am not too experienced with Docker. What do you mean by extracting library from container? The user should download my container at first and then I should use somehow libs from it? Any tutorials or materials will be appreciated :slight_smile:

Ah, no. I mean that for each of the architectures that lsquic supports (the 7 of them) you can have a Dockerfile that compiles them for the arch and then copy the final compiled artifact from inside of the container to the outside (meaning into your machine).

The point is, every time you want to do a release, you do this 7 times, extract the compiled libraries (.so / .dylib / etc.) and put them inside a .tar archive or some such, and put all those 7 links in your project’s GitHub Releases section.

When all of that works well, bonus points that you can actually donate Dockerfiles to lsquic. :slight_smile:

As for how you would do it in Docker, I could help you but can’t promise – waaaaay too busy. :frowning:

1 Like

I see now :slight_smile: But as far as I understand the user will be forced to download precompiled libs on his own using links I will put in hex.pm or I am missing something.
If yes then the user will be also forced to export some env variable so that my lib will know where to look for precompiled C sources.

There is proper tooling to compile C sources along your Erlang/Elixir sources but it’s error-prone. Still, there are projects doing it successfully. (Like Erlang’s esqlite, check it out for inspiration)

And yep, I was talking about manual installation if you’re going to precompile sources. Not universally applicable to everyone’s workflow, granted.

Lastly, why is Rust not an option?

Thanks for esqlite, I will take a look at it!

Regarding Rust, the goal is to build experimental wrapper around QUIC implementation so that you can use QUIC in Elixir and I have choosen C implementation mostly because I just wanted to learn programming in C :sweat_smile: I mean I’ve never never use/write any bigger project in C. I also wanted to use Unifex and Bundlex tools for connecting C and Elixir

1 Like

Oh, I see. So it’s mostly a learning project.

Well, good luck! :023:

1 Like