benoitc

benoitc

Erlang_python - Run Python across your Elixir cluster

Because it runs on the BEAM, you inherit Erlang/Elixir distribution for free.
Run Python on any node:

# Execute on remote node
:rpc.call(:"worker@host", :py, :call, [:numpy, :dot, [matrix_a, matrix_b]])

# Local call
{:ok, result} = :py.call(:sklearn, :fit, [model, data])

# Async task
{:ok, ref} = :py_event_loop.create_task(:heavy_compute, :run, [data])
# ... do other work ...
{:ok, result} = :py_event_loop.await(ref)

Key features:

  • Distributed by default via rpc:call
  • Async Task API (uvloop-inspired)
  • Channel API for bidirectional streaming
  • OWN_GIL subinterpreters (Python 3.12+)
  • Virtual environment management
  • ASGI/WSGI support

erlang_python 2.1 is out with async tasks and channel-based streaming.

Hex: erlang_python | Hex
GitHub:

Apache 2.0 licensed.

Most Liked

benoitc

benoitc

erlang_python embeds Python with true parallelism via sub-interpreters (each with its own GIL) or free-threading (no GIL). You get bidirectional communication: channels, async/await, and Python calling back into Erlang. Pythonx is designed for Livebook/Elixir with a single Python interpreter and shared GIL, which is enough for notebooks but limited for concurrent production workloads.

Afaik erlang_python is built to run Python applications inside the Erlang VM and inherit its efficiency.

Asd

Asd

Nice library!

I tried to read the code, but the project is huge and NIF part is around 15k lines of C code (thats around 5% of whole OTP C code size).

I think that it was generated, given that project is +90k lines in a month from you. Given that, it is strange to see that it is 2.1.0 version already. For me it feels more like its pre 1.0.0 version, given that it was rapidly developed and I assume was not used by anybody except you yet

Do you have any benchmarks? I am curious about how it compares to Snex, which uses erlang ports to spawn multiple interpreters and has very very tiny NIF surface. I think that erlang-python should be faster

Do you use it in production? What’s your use-case?

LostKobrakai

LostKobrakai

I’m curious about how this is different to pythonx?

Where Next?

Popular in Announcing Top

sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
985 42842 311
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43591 214
New
restlessronin
The repo is at GitHub - cyberchitta/openai_ex: Community maintained Elixir library for OpenAI API. Docs are at OpenaiEx User Guide — ope...
152 10095 134
New
Crowdhailer
I have been updating a library that allows you to pipe between functions that use the erlang result tuple convention. Assuming you have...
New
Qqwy
Today I realized that it would be possible to implement currying-capability in Elixir, using some clever anonymous function creation. ('c...
New
wojtekmach
Hey everyone! Req is an HTTP client for Elixir that I’ve been working on for quite some time. There is already a lot of HTTP clients out...
New
Azolo
Hey everyone, I just released WebSockex which is a Elixir WebSocket client. WebSockex strives to work as a OTP special process, be RFC6...
New
versilov
Could not wait for the missing Elixir ML libraries to appear, so, I wrote one myself, taking https://github.com/sdwolfz/exlearn as a foun...
New
Hal9000
Here is my first stab at this. README pasted below. https://github.com/Hal9000/elixir_random Comments and critiques are welcome. Th...
New
fuelen
Hey folks! Want to present a toolkit for writing command-line user interfaces. It provides a convenient interface for colorizing text...
New

Other popular topics Top

Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35953 110
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New

We're in Beta

About us Mission Statement