Snex - Easy and efficient Python interop for Elixir

Your Libraries & OS Mentoring Libraries
1

I’ve recently released v0.2.0 of my Python interop library, Snex.

This version rolls up all work-in-progress improvements that have been driven by half a year of use in a production Elixir system. Finally had some time to fully clean them up, and release a version that I’m proud of.

Snex is a library for interfacing with Python code in a tightly integrated way. The code is run by sidecar Python interpreters, managed by an Elixir runtime, and communicating through a light Snex Python runtime.

A big design goal of the library is to achieve devx similar to pythonx, while avoiding GIL-related issues and maintenance burdens of C-level integration - where both the BEAM and CPython APIs are constantly moving targets.

Highlights

  • Run multiple Python interpreters in separate OS processes, preventing GIL issues from affecting your Elixir application.

  • Leverages uv to manage Python versions and dependencies, embedding them into your application’s release for consistent deployments.

  • A powerful and efficient interface with explicit control over data passing between Elixir and Python processes.

  • Supports custom Python environments, asyncio code, and integration with external Python projects.

  • Built on stable foundations, so future versions of Python or Elixir are unlikely to require Snex updates to use - they should work day one!

Quick example

defmodule SnexTest.NumpyInterpreter do
  use Snex.Interpreter,
    pyproject_toml: """
    [project]
    name = "my-numpy-project"
    version = "0.0.0"
    requires-python = ">=3.10,<3.15"
    dependencies = ["numpy>=2"]
    """
end

{:ok, inp} = SnexTest.NumpyInterpreter.start_link()
{:ok, env} = Snex.make_env(inp)

{:ok, 6.0} =
  Snex.pyeval(env,
    """
    import numpy as np
    matrix = np.fromfunction(lambda i, j: (-1) ** (i + j), (s, s), dtype=int)
    """,
    %{"s" => 6},
    returning: "np.linalg.norm(matrix)")

Links

GitHub: GitHub - kzemek/snex: :snake: Easy and efficient Python interop for Elixir
Hex: snex | Hex
HexDocs: snex v0.2.0 — Documentation

16 Likes
2

Hi, very impressive library. I’ve read the code and it looks really great, however I have these questions and notes.

  1. You use NIF to manage envs. I’ve read it and it’s whole use-case is to send a port command during destruction to clean up the envs dict in interpreter. As far as I can see, it allows to pass these Env structures around in Elixir runtime so that when last instance of this resource is garbage collected, env will be removed from python interpreter. Very clever thing to do, I am impressed!
  2. If I send a command and die awaiting the result (for example command contained infinite async loop in python), it would still be stored in the Snex.Interpreter’s state. I don’t know about good workaround :sweat_smile:
  3. I find
     defp run_command(command, port) do
   id = :rand.bytes(16)
    Some kind of russian roulette, but with better chances :grin: . You’re calling comands from interpreter and just using an incrementing command_id_counter integer (with overflow) in it’s state would be much better than having a very little chance to break the code.
  4. In case of a bug (like infinite loop) in init script, Snex.Interpreter will forever stuck in receive in init_python_port function. I’d suggest to add a timeout as a failsafe
  5. As far as I can see, it is not possible pass an arbitrary object to Elixir to be later used in subsequent pyeval calls, right? But it is possible to setup some env, execute a command in it setting a variable and this variable will be set in this env (with the same ID, I mean), right?
  6. Are you planning to implement interruptions of the evaluations? For example, some process does pyeval with async code and dies. It would make sense for interpreter to stop this evaluation in such situation
  7. Have you considered implementing a C-node interface in python? It’s much more work than your implementation, but it would provide a more straightforward integration (I think)

Overall, very good and clever code, I will definitely recommend it to my friends and colleagues!

1 Like
3

Thanks for the notes, much appreciated!

Right. However, I see tasks holding on to their environments before they finish as a feature, not an edge case. For example, you can start an async task from Elixir and immediately discard your env:

Snex.pyeval(env, """
  import asyncio
  loop = asyncio.get_event_loop()
  loop.call_soon(some_async_task)
  """)

Inside, some_async_task can even send messages back to BEAM processes with snex.send(), so starting tasks without blocking is a useful pattern for Snex.

You’re right, in the current iteration it could be an incrementing integer. At an early point in the design, I had other processes sending commands directly to the port, so ID generation was decentralized. I still want to have that option open for the future.

16 random bytes is around the same entropy as UUIDv4, so I defaulted to that. Now that I think about it, it could also be an :erlang.unique_integer() instead for 64 bits of space, although shared with other users on the node.

Good idea!

Correct. The usual pattern would be exactly to prepare an env and repeatedly reference the object from the env. With various opts of make_env/3 you can also copy Python objects between the envs.

I think it makes sense to be able to list running tasks and cancel them from Elixir side. Even a task stuck in an infinite loop with no awaits might be cancellable using OS signals.

If you search around for “elixir python” or “erlang python”, there’s a lot of projects that tackled this through C-level integration and are unusable today as they required constant maintenance to stay “fresh”. PyErl was one such project implementing C-node Python.

Making a C-node out of Python is less of a maintenance burden than directly integrating CPython API like pythonx does, but it still requires tracking Erl_Interface which is changing between BEAM versions, sometimes in breaking ways.

An important design goal of Snex is to stay evergreen as much as possible, so the current version is likely to still be usable with Python 3.17/Elixir 1.21/Erlang 30.

Thank you!

3 Likes