Snex - Easy and efficient Python interop for Elixir

Hi, very impressive library. I’ve read the code and it looks really great, however I have these questions and notes.

1. You use NIF to manage envs. I’ve read it and it’s whole use-case is to send a port command during destruction to clean up the envs dict in interpreter. As far as I can see, it allows to pass these Env structures around in Elixir runtime so that when last instance of this resource is garbage collected, env will be removed from python interpreter. Very clever thing to do, I am impressed!
2. If I send a command and die awaiting the result (for example command contained infinite async loop in python), it would still be stored in the Snex.Interpreter’s state. I don’t know about good workaround
3. I find

    defp run_command(command, port) do
      id = :rand.bytes(16)
   

   Some kind of russian roulette, but with better chances . You’re calling comands from interpreter and just using an incrementing command_id_counter integer (with overflow) in it’s state would be much better than having a very little chance to break the code.
4. In case of a bug (like infinite loop) in init script, Snex.Interpreter will forever stuck in receive in init_python_port function. I’d suggest to add a timeout as a failsafe
5. As far as I can see, it is not possible pass an arbitrary object to Elixir to be later used in subsequent pyeval calls, right? But it is possible to setup some env, execute a command in it setting a variable and this variable will be set in this env (with the same ID, I mean), right?
6. Are you planning to implement interruptions of the evaluations? For example, some process does pyeval with async code and dies. It would make sense for interpreter to stop this evaluation in such situation
7. Have you considered implementing a C-node interface in python? It’s much more work than your implementation, but it would provide a more straightforward integration (I think)

Overall, very good and clever code, I will definitely recommend it to my friends and colleagues!

Thanks for the notes, much appreciated!

Right. However, I see tasks holding on to their environments before they finish as a feature, not an edge case. For example, you can start an async task from Elixir and immediately discard your env:

Snex.pyeval(env, """
  import asyncio
  loop = asyncio.get_event_loop()
  loop.call_soon(some_async_task)
  """)

Inside, some_async_task can even send messages back to BEAM processes with snex.send(), so starting tasks without blocking is a useful pattern for Snex.

You’re right, in the current iteration it could be an incrementing integer. At an early point in the design, I had other processes sending commands directly to the port, so ID generation was decentralized. I still want to have that option open for the future.

16 random bytes is around the same entropy as UUIDv4, so I defaulted to that. Now that I think about it, it could also be an :erlang.unique_integer() instead for 64 bits of space, although shared with other users on the node.

Good idea!

Correct. The usual pattern would be exactly to prepare an env and repeatedly reference the object from the env. With various opts of make_env/3 you can also copy Python objects between the envs.

I think it makes sense to be able to list running tasks and cancel them from Elixir side. Even a task stuck in an infinite loop with no awaits might be cancellable using OS signals.

If you search around for “elixir python” or “erlang python”, there’s a lot of projects that tackled this through C-level integration and are unusable today as they required constant maintenance to stay “fresh”. PyErl was one such project implementing C-node Python.

Making a C-node out of Python is less of a maintenance burden than directly integrating CPython API like pythonx does, but it still requires tracking Erl_Interface which is changing between BEAM versions, sometimes in breaking ways.

An important design goal of Snex is to stay evergreen as much as possible, so the current version is likely to still be usable with Python 3.17/Elixir 1.21/Erlang 30.

Thank you!

Happy new year! Snex 0.3 is out with some major upgrades. The biggest highlights are the revamped serialization story and await snex.call(m, f, a) in Python, but there’s been a lot of smaller enhancements all around.

Below is the abbreviated changelog. The full one can be found in CHANGELOG.md, and details on current usage patterns can be found in the updated README.md - I find the Encoding/decoding table particularly interesting!

Highlights

• Improved serialization

  Serialization protocol between Python & Elixir has been changed.
  Instead of JSON encoding both ways, Elixir now encodes terms into a restricted Pickle (v5) format, while Python encodes objects into a restricted External Term Format.
  The encoded data is then decoded with pickle.loads() on Python side, and :erlang.binary_to_term/1 in Elixir.

  Writing only encoders in both languages allows the implementation to be small and portable between versions, and reusing native decoding routines makes it highly performant - especially in Python.

  Consequently:

  • tuples are now encoded as tuples instead of lists
  • all Elixir terms can be encoded and round-tripped (fallback to :erlang.term_to_binary/1)
  • all Python basic objects can be encoded
  • floats preserve representation between languages
  • encoding customization is available on both sides

• New ways of calling Elixir code from Python

  Code running under Snex can use snex.call(m, f, a) and snex.cast(m, f, a) to communicate with the BEAM.
  snex.call can be awaited on, and will return the result of the call, while snex.cast is fire-and-forget (returns None).

• Sigils for code location

  import Snex.Sigils to use ~p"code", ~P"code" sigils, which create %Snex.Code{} automatically.

  iex> Snex.pyeval(env, ~p"raise RuntimeError('test')")
  
  {:error,
    %Snex.Error{
    traceback: [...,
      "  File \"/Users/me/snex/CHANGELOG.md\", line 41, in <module>\n    raise RuntimeError(\"test\")\n",
      "RuntimeError: test\n"]
    }}
  

• New Snex.Interpreter (and custom interpreters) options

  • :label - labels the interpreter process through :proc_lib.set_label/1.

  • :init_script - now also takes {script, args} tuple, to pass variables to the init script and the root environment it prepares.

  • :init_script_timeout - if :init_script doesn’t finish under the timeout, the interpreter process stops with %Snex.Error{code: :init_script_timeout}.

  • :wrap_exec - customizes how the Python process is spawned by wrapping the executable path and arguments.
    This can be used e.g. to run Python inside a Docker container or set it up with cgroups.

• New functions

  • Snex.destroy_env/1 - explicitly cleans up the referenced Python environment
  • Snex.Env.disable_gc/1 - opts out of automatic lifetime management for a %Snex.Env{}
  • Snex.Env.interpreter/1 - gets the interpreter process associated with a %Snex.Env{}
  • Snex.Interpreter.os_pid/1 - gets the OS PID of the Python interpreter process.
  • Snex.Interpreter.stop/1,2,3 - stops the interpreter process

• Move serde work to Snex callers

  Serialization and deserialization on Elixir side is now done outside of Snex.Interpreter process.

• Move Snex interface code to a public module snex

  You can now import snex module from your external Python code to get awareness of Snex Python-side types and interface.

Snex 0.4.0 is out after a few weeks of RC production use! This release focuses on performance and devx, and support for using Snex from Python subprocesses.

As before, below is the abbreviated changelog with some of the more interesting changes. The full one can be found in CHANGELOG.md. Changelog entries are written by hand, so it should be very readable.

Highlights

• Improved performance

  Simple Snex.pyeval call latency dropped from 75 µs (v0.3.2) to 38 µs end-to-end on a MacBook M1 - nearly 2× lower. There’s a clear path to 28 μs, slated for v0.4.1.

• snex Python module documentation

  snex Python interface is now documented with ExDoc!

Breaking changes

• Snex.pyeval/4 no longer returns bare :ok

  Snex.pyeval/4 now always returns either {:ok, value} or {:error, reason}.

• snex.send and snex.cast now require await

  The write transport from Python to Elixir is now under standard Python asyncio flow control. As a side effect, send()&cast() were required to become async.

Features

• Snex.pyeval/4 can execute code with Python return statements

  The :returning option is now deprecated - just write Snex.pyeval(env, "return 42")

• Multiprocessing support

  snex.serve and snex.io_loop_for_connection provide a clean way to connect Python subprocesses - or any external processes - to the Snex system, enabling snex.call() & snex.cast() from outside of the main Python process.

• snex.Elixir proxy object

  Added syntax sugar for snex.cast()/snex.call() for a natural Elixir-like call syntax in Python:

  await Elixir.Enum.frequencies(["a", "b", "a", "a", "d", "b"])
  

• snex.LoggingHandler

  Snex Python interface now provides a snex.LoggingHandler class for use with Python’s logging. The handler outputs Python logs using Elixir’s Logger.

• Snex.pyeval/4 can be called with interpreter instead of %Snex.Env{}

  Simplifies setup and improves latency for one-off commands.

https://github.com/kzemek/snex

Snex 0.4.1 is out now!

This release introduces an eager asyncio loop improvement discussed on the Python forum. This brings the end-to-end latency of a simple Snex.pyeval down to 28 μs on CPython, 20 μs on PyPy - closing in on NIF-based integrations. For comparison, an equivalent Pythonx.eval+decode is ~6.5 μs^[1].

I also revamped the Highlights section of README.md to better showcase what Snex is and can do. I’m putting it down below for a refresher.

Thanks!

Highlights

Robust & Isolated - Run any number of Python interpreters in separate OS processes, preventing GIL issues or blocking computations from affecting your Elixir application. You can call asyncio code, use PyPy instead of CPython, or even run Python in a Docker container!

Declarative Environments - Leverages uv to manage Python versions and dependencies, embedding them into your application’s release for consistent deployments. Supports custom Python environments and easy integration with Python projects.

Bidirectional communication - Powerful and efficient interface with explicit control over data. Python code running under Snex can send messages to BEAM processes and call Erlang/Elixir functions.

High quality, organic code - Every line of Snex is thought out and serves a purpose. Code is optimized to keep performance overhead low.

Forward Compatibility - Built on stable foundations independent of C-level interfaces, so future versions of Python and Elixir will work on day one!

https://github.com/kzemek/snex

1. in local serialized benchmarks on an M1 Macbook Pro. The latency story is completely different in concurrent usage, where Pythonx gets blocked on GIL. ↩︎

“snex” - love the name

Snex 0.4.2 is out! This patch release contains a fix that works around a BEAM Port resource leak, which can permanently stall communication through a port.