LegitStack

LegitStack

I have this weird idea. I want to use elixir as a messaging service between python functions across a network.

Luckily, I found this post which explains how to use erlport to call python functions from elixir.

One use case for this sort of thing is as a worker-scheduler type system. But more generally it seems like a cool way to implement an actor system for python - just use an actor language.

My question to you guys is: what’s wrong with this idea? Is it doable? is it doable but difficult?

Keep in mind I’ve never really used erlang or elixir, but I love ideas behind the language. So what might I be missing?

Just so we’re on the same page let me give you a concrete example of how a framework like this could be used:

Say you have a network with 5 computers. 4 powerful machines we’ll call the ‘cluster’ and 1 laptop, we’ll call the ‘scheduler.’

You could start up a genserver (or something, I barely know what that is) so that there are workers on the clusters waiting to pull tasks from some elixir process on the scheduler. The idea is that the users (who only knows python, because lets say they are datascientists) would send a list of python functions to the scheduler using erlpoint, then the workers would pull those functions and execute the python call using erlport until all the tasks were depleted.

Of course, this is not the only use case, but its a simple, functional one.

What do you think?

Showing Posts 1 to 8

Nicd

Nicd

Your example sounds a lot like what Celery already does well in Python land. It uses RabbitMQ by default to deal with the task distribution.

Fl4m3Ph03n1x

Fl4m3Ph03n1x

Usually, if you can just get away with Python without using erlang/elixir, that would be the recommended way as it would make your stack smaller and easier to deal with.

With this said, I believe your idea is doable. But this isn’t really saying much. Elixir/Erlang are quite powerful, most things are doable, that is not the question. The question is - would it be efficient and easyhttps://www.youtube.com/watch?v=CmMMpaUD3g8 enough for your use case?

And here I believe that with the right architecture, it would. But you would have to pay a heavy price in complexity.

For starters, you are talking about clusters of machines with Nodes. Erlang and Elixir support inter-Node communication by default, but don’t be fooled, that is a really hard thing to get right. Problems like maintaining a coherent state and knowing what to do when a Split Brain happens are just 2 of the challenges you likely will have to deal with - and they are not exactly a walk in the park to deal with.

Then there are other things like “What happens if the master Node (the one who has all the tasks) dies? From what Node will workers pull tasks from?” (Election algorithm incoming!)

Overall, if you want several machines to communicate, you’d have to deal with a lot of distributed systems challenges.

To this effect, you could go down the road of implementing your own solutions Raft comes to mind as a possible consensus algorithm and you can investigate more on how the folks at Phoenix managed to fix similar issues (https://phoenixframework.org/).

By all means, if you can get away by doing things in a single Node, go ahead, things like Flow already exist so you can take the most out of any machine.

If not, then you will have another level of challenges to fight and at least I won’t be able to suggest much more.

Now, if you already have solutions for these complex issues (or simply decide you don’t want to solve them for now) then Elixir really has a nice support for distributed computing with its primitives, and there is a lot of information out there to help you get started! Here are some examples and talks!

The important thing is to not be afraid. Hope it helps!

LegitStack

LegitStack OP

thanks, for the overview and the references to other resources, it has already helped me understand what would be required better!

LegitStack

LegitStack OP

Your example sounds a lot like what Celery already does well in Python land. It uses RabbitMQ by default to deal with the task distribution.

Indeed, and there are other solutions too, like Dask which is fully contained in python. Elixir just seems more pure for handling distributed systems than python, which is where the impetus for this idea came from.

peerreynders

peerreynders

I think the fundamental question is: what real world problem is being solved by your idea?

Python evolved to solve certain problems in a certain manner in a given environment.

The BEAM with it’s process based nature approaches real world problems in a very different way - a way which is fundamentally at odds with the way libraries implemented in Python (and the lower level C code) operate.

Another Python integration technology which hasn’t been mentioned yet: Pyrlang

bottlenecked

bottlenecked

Hi there, to me this sounds like a pub/sub problem that could be solved using a python lib build on top of pika (the python rabbitmq client lib) to model your domain, and the rest is messaging and queues using rabbitmq. Your consumer services could also be written in python then.

I wouldn’t go down the distributed-nodes-in-elixir-path, it seems overly complex- unless you’re doing it for learning purposes.

OvermindDL1

OvermindDL1

I actually made an actor framework for stackless python over 10 years ago, it worked very well, and even had features like being able to serialize an entire actor and move it around, store it, reload it, etc… ^.^

Normal python is missing some features that make it easy though.

kpanic

kpanic

Not sure what you want to do but GitHub - celery/kombu: Messaging library for Python. · GitHub powers celery, you might want to have a look into it to build your own “custom” workers
Since you are curious about erlang/elixir, you might want to try also GitHub - Pyrlang/Pyrlang: Erlang node implemented in Python 3.5+ (Asyncio-based) · GitHub

NOTE that Ask Solem, the author of celery inspired herself from erlang to create celery/kombu.
That’s one of the reasons that pushed me to try/learn erlang/elixir

— All posts loaded —

Where Next? Top

Trending in Questions Top

Hellowman
Hello everyone! I am new to coding with Elixir/Erlang. And i am looking for some utilities for fuzzing testing like AFL++. Does anyone ...
New
yarrichar
I’m very new to Elixir, so sorry if this is a silly question: I am wondering what happens to the mailbox of a process when the machine i...
New
pragdave
How do I tell Hologram not to escape HTML when substituting into templates? {Textile.render(@product.long_description)} escapes the HTM...
New
Fl4m3Ph03n1x
Background When trying to execute mix release on a Windows 11 machine for a Phoenix project I get the following error: * assembling mark...
New
mandar1045
I load tested a plain Phoenix chat app backed by Postgres with 100 million messages in it, everything on default settings, to see what br...
New
rlipscombe
I’d like to create a file with 600 permissions (rw-------), but there’s a race condition if I create it and then change the permissions. ...
New
saleyn
Is there a way to import record definitions in the iex shell for pretty printing (similar to the rr command in the erl shell)? This woul...
New

Other Trending Topics Top

GenericJam
Edit: 2026 May 15 - This post is archived. Mob is alive!! Main docs: mob v0.7.11 — Documentation A bit of explanation for the slightly c...
261 13183 112
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews