Zigler: zig nifs for Elixir

Your Libraries & OS Mentoring Libraries
, ,
1

Currently just starting out on a new mini-project - getting zig NIFs to run in elixir.

The idea here is to make the zig NIFs be “embedded” in the elixir code, much like an “asm” call in C. It also makes the bridge between the zig function call and the elixir “basically transparent”. Several adapters are provided: i64, f64, string (as a c string or a zig slice), i64 arrays, f64 arrays. Of course it’s not terribly hard to do the unmarshalling of the erlang terms yourself inside the NIF.

Haven’t started documenting yet, but if you want to see how it works, the test/ directory has quite a few instructive examples which are all passing as of zig 0.4.0

Comments and criticism appreciated!

35 Likes
2

Lol, that’s cool. ^.^

1 Like
3

Thanks! That means a lot to me. I think zig is a great fit for elixir. Incidentally, do you know of any good references on refrences? I’m a little bit blocked on my understanding of them.

Edit:. Oops, resources, not references

4

Very cool project…

5

Do you mean OTP resources? Where you can store native code specific data pointers and so forth? Or something in Zig?

6

Yep, otp resources. I’m trying to get my head around what the right way is to square them with zig’s opinionated (but optional) allocator semantics. One thing I’m unsure about is if otp resources can be realloc’d, since there is no resource realloc function. Probably not hard to try though, once I figure out how to even make one.

7

I would just have you allocate a resource with enough space to hold a zig pointer and store that pointer in it. Have the resource deallocation function deallocate the zig object (if you are done with it of course). That’s generally what I do regardless, I rarely have resource be ‘big’, I treat them as basically just a pointer to my internal object or pointer.

So no, you can’t realloc them, treat them as handles and you are good to go.

1 Like
8

When I try this out locally I don’t seem to have the mix task.

  defp deps do
    [
      {:zigler, git: "https://github.com/ityonemo/zigler.git"}
      # {:dep_from_hexpm, "~> 0.3.0"},
      # {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", tag: "0.1.0"}
    ]
  end

Then I ran mix do deps.get, deps.compile

Then

> mix zigler.get_zig latest
Compiling 1 file (.ex)
Generated zig_test app
** (Mix) The task "zigler.get_zig" could not be found

Any clue what I’m doing wrong here?

Edit it seems like only Linux is supported atm and I’m running macOS

9

that’s super strange, it should work though (just tested the mix task myself on a completely fresh system).

ityonemo@hadamard:~/code$ git clone https://github.com/ityonemo/zigler
Cloning into 'zigler'...
remote: Enumerating objects: 176, done.
remote: Counting objects: 100% (176/176), done.
remote: Compressing objects: 100% (108/108), done.
remote: Total 176 (delta 80), reused 148 (delta 52), pack-reused 0
Receiving objects: 100% (176/176), 33.32 KiB | 897.00 KiB/s, done.
Resolving deltas: 100% (80/80), done.
ityonemo@hadamard:~/code$ cd zigler
ityonemo@hadamard:~/code/zigler$ mix zigler.get_zig latest
Unchecked dependencies for environment dev:
* mojito (Hex package)
  the dependency is not available, run "mix deps.get"
** (Mix) Can't continue due to errors on dependencies
ityonemo@hadamard:~/code/zigler$ mix deps.get
Resolving Hex dependencies...
Dependency resolution completed:
Unchanged:
  castore 0.1.3
  mint 0.4.0
  mojito 0.5.0
  poolboy 1.5.2
* Getting mojito (Hex package)
* Getting castore (Hex package)
* Getting mint (Hex package)
* Getting poolboy (Hex package)
ityonemo@hadamard:~/code/zigler$ mix zigler.get_zig latest
===> Compiling poolboy
==> castore
Compiling 1 file (.ex)
Generated castore app
==> mint
Compiling 1 file (.erl)
Compiling 22 files (.ex)
Generated mint app
==> mojito
Compiling 14 files (.ex)
Generated mojito app
==> zigler
Compiling 5 files (.ex)
Generated zigler app

09:13:22.758 [info]  downloading zig version 0.5.0 and caching in /home/ityonemo/code/zigler/zig.
ityonemo@hadamard:~/code/zigler$ mix test
Compiling 5 files (.ex)
Generated zigler app
...................................

Finished in 4.9 seconds
35 tests, 0 failures

In theory there is no reason why most of this stuff shouldn’t work on OSX, but I need to get a few paths correct, but i have no way to test it. If you would like to try getting it to work on OSX, any help would be appreciated! In the meantime I’m going to put a warning about OS at build time.

1 Like
10

Yeah, I got it almost working int this test PR https://github.com/ityonemo/zigler/pull/16. It seems like the mix task had a hardcoded check for :os.system. But you’re right, it seems the only real blocker is a path issue.

11

ok, I will try to generalize the operating system when i get a chance to work on this tonight, will let you know how it goes.

2 Likes
12

Awesome! If it would be helpful, I’ll happily test things out on my system.

1 Like
13

I would just have you allocate a resource with enough space to hold a zig pointer

lol i can do that? Wow. Thanks! This will be relatively easy.

1 Like
14

This looks really cool! Looks like I’ll have to add this to the todo list of things to play with alongside Rustler

2 Likes
15

I’m going to 0.1.0 on Hex.pm by the end of the month. Currently I’m working on three last major features - seamless integration of zig tests into ExUnit, translation of compiler warnings to reflect the actual location of code, and assembly of zig docstrings into full zig documentation as a part of ExDoc. =D This might take some time since I need to revamp the parser to be not a terrible ad-hoc solution.

However aside from the QoL features it basically “just works”, so I’d suggest checking it out now. You’ll have a much easier time getting up and running with zigler than rustler since in zigler you literally write your zig code inside the elixir.

Also having eyes on MacOS and an expert in Erlang Windows would be awsome, since I don’t have the capability to test those platforms ATM.

3 Likes
16

This looks really interesting!

I wonder if you’ve considered creating a zigler mix compiler? That way we could write .zif files that are compiled as an alternative to sigils? I wonder if that might feel more natural?

17

Why not both?

I do like the idea of it being like the c “asm” keyword, you can call out zig file dependencies with zig’s @include directive if you need more code than you’d like to ship (and you should do that anyway) (in this case, path relative to the elixir module) and currently “it just works”. The nice thing about making it module local is that it carries with it inferrable information about how to bind it in using erlang’s nif adapters, which you’d have to manually configure if you lose the association with an elixir module.

I probably should also give a common location setting to draw files from.

18

Sorry if I implied replacing your currently strategy. I mean “in addition to”. @include sounds a good half-way house but I wouldn’t suggest my opinion carries any weight.

1 Like
19

Version 0.1.0 is out.

https://hexdocs.pm/zigler/Zigler.html

Features:

  • ~Z used as an inline entry point for the NIF: all function bindings are inferred from these entry points and correctly and automatically marshalled into the erlang NIF template, including type checking and conversion on ingress and type conversions on egress, and correct @spec for static type checking.
    Example:
defmodule ExampleTest do
  use ExUnit.Case, async: true
  use Zigler, app: :zigler

  ~Z"""
  /// nif: i64_list_in/1
  fn i64_list_in(val: []i64) i64 {
    var total: i64 = 0;
    for (val) |item| {
      total += item;
    }
    return total;
  }
  """
  describe "i64 lists can be ingressed" do
    test "correctly" do
      assert 6 == i64_list_in([1, 2, 3])
    end
  end
end

  • Compilation assistance: If you should encounter a Zig compiler error, the compiler will redirect you to the correct file/line (including if it’s inside a ~Z block)

  • integration of erlang’s native alloc/realloc/free with zig’s composable allocator idiom

  • Full documentation integration with ExDoc (for example, this doc file is entirely created from Zig docstrings: https://hexdocs.pm/zigler/beam.html#content).

  • Unit test integration with ExUnit.

  • Ability to statically or dynamically bind external libraries with the C ABI
    Example (from the unit tests):

defmodule BlasStatic do
      use Zigler, app: :zigler, libs: ["/usr/lib/x86_64-linux-gnu/blas/libblas.a"]

      ~Z"""
      const blas = @cImport({
        @cInclude("/usr/include/x86_64-linux-gnu/cblas.h");
      });

      /// nif: blas_axpy/3
      fn blas_axpy(env: beam.env, a: f64, x: []f64, y: []f64) beam.term {

        if (x.len != y.len) {
          return beam.throw_function_clause_error(env);
        }

        blas.cblas_daxpy(@intCast(c_int, x.len), a, &x[0], 1, &y[0], 1);

        return beam.make_f64_list(env, y) catch {
          return beam.throw_function_clause_error(env);
        };
      }
      """
    end

    test "we can use statically-linked blas" do
      # returns aX+Y
      assert [11.0, 18.0] == BlasStatic.blas_axpy(3.0, [2.0, 4.0], [5.0, 6.0])
    end
5 Likes
20

One last thing, I’ll be giving a talk about this at the San Francisco Elixir/erlang meetup.in January if you’d like to ask questions or learn more in person

3 Likes