benoitc
Erlang_python - Run Python across your Elixir cluster
Because it runs on the BEAM, you inherit Erlang/Elixir distribution for free.
Run Python on any node:
# Execute on remote node
:rpc.call(:"worker@host", :py, :call, [:numpy, :dot, [matrix_a, matrix_b]])
# Local call
{:ok, result} = :py.call(:sklearn, :fit, [model, data])
# Async task
{:ok, ref} = :py_event_loop.create_task(:heavy_compute, :run, [data])
# ... do other work ...
{:ok, result} = :py_event_loop.await(ref)
Key features:
- Distributed by default via rpc:call
- Async Task API (uvloop-inspired)
- Channel API for bidirectional streaming
- OWN_GIL subinterpreters (Python 3.12+)
- Virtual environment management
- ASGI/WSGI support
erlang_python 2.1 is out with async tasks and channel-based streaming.
Hex: erlang_python | Hex
GitHub:
Apache 2.0 licensed.
Most Liked
benoitc
erlang_python embeds Python with true parallelism via sub-interpreters (each with its own GIL) or free-threading (no GIL). You get bidirectional communication: channels, async/await, and Python calling back into Erlang. Pythonx is designed for Livebook/Elixir with a single Python interpreter and shared GIL, which is enough for notebooks but limited for concurrent production workloads.
Afaik erlang_python is built to run Python applications inside the Erlang VM and inherit its efficiency.
Asd
Nice library!
I tried to read the code, but the project is huge and NIF part is around 15k lines of C code (thats around 5% of whole OTP C code size).
I think that it was generated, given that project is +90k lines in a month from you. Given that, it is strange to see that it is 2.1.0 version already. For me it feels more like its pre 1.0.0 version, given that it was rapidly developed and I assume was not used by anybody except you yet
Do you have any benchmarks? I am curious about how it compares to Snex, which uses erlang ports to spawn multiple interpreters and has very very tiny NIF surface. I think that erlang-python should be faster
Do you use it in production? What’s your use-case?
Popular in Announcing
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Courses that'll move you from confusion to "Aha, now I get it!"