benoitc
Erlang_python - Run Python across your Elixir cluster
Because it runs on the BEAM, you inherit Erlang/Elixir distribution for free.
Run Python on any node:
# Execute on remote node
:rpc.call(:"worker@host", :py, :call, [:numpy, :dot, [matrix_a, matrix_b]])
# Local call
{:ok, result} = :py.call(:sklearn, :fit, [model, data])
# Async task
{:ok, ref} = :py_event_loop.create_task(:heavy_compute, :run, [data])
# ... do other work ...
{:ok, result} = :py_event_loop.await(ref)
Key features:
- Distributed by default via rpc:call
- Async Task API (uvloop-inspired)
- Channel API for bidirectional streaming
- OWN_GIL subinterpreters (Python 3.12+)
- Virtual environment management
- ASGI/WSGI support
erlang_python 2.1 is out with async tasks and channel-based streaming.
Hex: erlang_python | Hex
GitHub:
https://github.com/benoitc/erlang-python
Apache 2.0 licensed.
Most Liked
benoitc
erlang_python embeds Python with true parallelism via sub-interpreters (each with its own GIL) or free-threading (no GIL). You get bidirectional communication: channels, async/await, and Python calling back into Erlang. Pythonx is designed for Livebook/Elixir with a single Python interpreter and shared GIL, which is enough for notebooks but limited for concurrent production workloads.
Afaik erlang_python is built to run Python applications inside the Erlang VM and inherit its efficiency.
Asd
Nice library!
I tried to read the code, but the project is huge and NIF part is around 15k lines of C code (thats around 5% of whole OTP C code size).
I think that it was generated, given that project is +90k lines in a month from you. Given that, it is strange to see that it is 2.1.0 version already. For me it feels more like its pre 1.0.0 version, given that it was rapidly developed and I assume was not used by anybody except you yet
Do you have any benchmarks? I am curious about how it compares to Snex, which uses erlang ports to spawn multiple interpreters and has very very tiny NIF surface. I think that erlang-python should be faster
Do you use it in production? What’s your use-case?
Last Post!
gtcode
This is really cool — there’s a lot of Python+Elixir crossover happening right now.
I built a Python FFI for Elixir: https://github.com/nshkrdotcom/snakebridge
Then put together a sample project to test whether pooling Python workers from the BEAM side buys anything over straight Python. Turns out non-GIL Python is pretty capable on its own: https://github.com/nshkrdotcom/slither
That said, the whole stack is still a prototype — glued together with gRPC and architecturally rough. Feel free to steal anything useful. Really glad to see your project.
Popular in Announcing
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"