kaquadu

kaquadu

Hello!
I am writing my thesis application. I need linear programming, but my app is written in Elixir, which is really not the language for such operations. That is why I decided to use Erlport as the Elixir dependency, which is capable of connecting Python code with Elixir. I’m also using Pulp as the python library for the optimization.

Elixir version: 1.10.4,
Erlport version: 0.10.1,
Python version: 3.8.5,
PuLP version: 2.3

I’ve written such a module for Elixir-Python communication, which leverages the GenServer as the main ‘communication hub’ between Elixir and Python:

defmodule MyApp.PythonHub do
  use GenServer

  def start_link(_) do
    GenServer.start_link(__MODULE__, nil, name: __MODULE__)
  end

  def init(_opts) do
    path = [:code.priv_dir(:feed), "python"]
          |> Path.join() |> to_charlist()

    {:ok, pid} = :python.start([{ :python_path, path }, { :python, 'python3' }])

    {:ok, pid}
  end

  def handle_call({:call_function, module, function_name, arguments}, _sender, pid) do
    result = :python.call(pid, module, function_name, arguments)
    {:reply, result, pid}
  end

  def call_python_function(file_name, function_name, arguments) do
    GenServer.call(__MODULE__, {:call_function, file_name, function_name, arguments}, 10_000)
  end

end

The GenServer module is calling python file, which contains such a function:

def calculate_meal_4(products_json, diet_json, lower_boundary, upper_boundary, enhance):
  from pulp import LpMinimize, LpProblem, LpStatus, lpSum, LpVariable, value
  import json
  products_dictionary = json.loads(products_json)
  print(products_dictionary)
  diets_dictionary = json.loads(diet_json)
  print(diets_dictionary)

  model = LpProblem(name="diet-minimization", sense=LpMinimize)

  # ... products setup ...

  x = LpVariable("prod_1_100g", lower_boundary, upper_boundary)
  y = LpVariable("prod_2_100g", lower_boundary, upper_boundary)
  z = LpVariable("prod_3_100g", lower_boundary, upper_boundary)
  w = LpVariable("prod_4_100g", lower_boundary, upper_boundary)

  optimization_function = # ... optimization function setup ...

  model += # ... optimization boundary function setup ...

  model += optimization_function

  print(model)

  solved_model = model.solve()

  print(value(model.objective))

  return [value(x), value(y), value(z), value(w)]

The call to the GenServer itself looks like that:

PythonHub.call_python_function(:diets, python_function, [products_json, meal_statistics_json, @min_portion, @max_portion, @macro_enhancement])

where python_function is :calculate_meal_4 and products_json and meal_statistic_json are jsons containing required data.

While calling calculate_meal_4 via python3 diets.py, which launches the python script above with some example, but real (taken from the app), data everything works fine - I’ve got the minimized result in almost no time. The problem occurs while calling the python script via Elixir Erlport. Looking at the printed outputs I can tell that it seems working until

solved_model = model.solve()

is called. Then the script seems to freeze and GenServer finally reaches the timeout on GenServer.call function.

I’ve tested also the call on a simple python test file:

def pass_var(a):
  print(a)
  return [a, a, a]

and it worked fine.

That is why I am really consterned right now and I am looking for any advices. Shamefully I found nothing yet.

Showing Posts 1 to 2

kaquadu

kaquadu OP

With some help from Stack Overflow I’ve managed to solve this problem by making .py file executable and calling it via System.cmd - more info: Stack Overflow Thread

aseigo

aseigo

There are a couple of ways to deal with this:

a) call is sync .. but can wait forever. Instead of using the default timeout, pass in :inifinity as the timeout param to the GenServer and then just … wait. :slight_smile:

b) call is sync .. so don’t use it. Use cast instead and make the Elixir side of the code properly async around this: wrap the Python call in a Task, use message passing to get results around. However, this is not really a great option in this case due to the python process being stateful and synchronous.

c) call is sync .. but can return a ref instead of the answer right away and then return the actual answer later on. This pushes the async’ness into the GenServer (easy path: wrap the python call in a Task), and is accomplished by returning {:noreply, new_state} from the handle_call implementation and when the python finishes returning the result with GenServer:reply/2. Suffers from the same issue as (b)

d) Don’t use a GenServer in these sorts of cases! Just use a regular module and call its functions as needed. This means having a setup, use, and cleanup set of functions which would need to be used by callers. Not as pretty at the call-sites, but gets rid of the GenServer business. Not an amazing solution if you are doing a lot of calls into your python as that python setup time can be an expensive bit.

You can also consider using a pool of python-instance GenServers which are used to service calls .. with a pool of e.g. 10 python instances a simple API to make python calls could be provided which checks out an available server (and either waits until one is available, potentially for quite a while, or returns with a timeout if the pool is depleted for an extended period of time, if that makes sense in your application), runs the command, and returns when the results are available. This has the benefit of giving your application some concurrency for python calls by spreading them out across multiple environments .. but assumes that each call is separate and does not rely on state being held in the python process between subsequent calls.

(I’ve used erlport to perform long-running (and stateful, even..) ML workloads via python from distributed Elixir applications before, so this all sounded rather familiar :wink: )

— All posts loaded —

Where Next? Top

Trending in Questions Top

RSP87
I’m working on a project that simulates the bumbl example in the programming phoenix book. It acts almost like an email client. We have a...
New
kpanic
Hi everyone, I am toying with the idea of building a “match maker” for giving personal help to people that wants to start coding. I sta...
New
nseaSeb
Hello, I know there is an approach for handling lists that allows for optimized traversal, but I can’t recall the specific method (somet...
New
brecabral
Documentation While reading the Scoped Routes section, I noticed that the documentation currently refers to a problem without explainin...
New
velrest
So my question is quite simple and i have found no conclusive answer on forum, google or AI. Should we use :erlang.float for Integer to ...
New
asweet-confluent
I recently noticed that Elixir’s Logger defaults its primary log level to :debug when no :logger, :level application configuration is pre...
New
apz
I’m new to elixir and just tried to install the elixirLS extension for VScode(ium) and it is throwing some errors that I would like help ...
New

Other Trending Topics Top

GenericJam
Edit: 2026 May 15 - This post is archived. Mob is alive!! Main docs: mob v0.7.11 — Documentation A bit of explanation for the slightly c...
New
JesseHerrick
Hey, I’m Jesse and I’m the main contributor behind Dexter, a full-featured, lightning-fast Elixir LSP optimized for large codebases. It s...
New
mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
marciok
Hi there! We created Gust: A task orchestrator inspired by Airflow. For those who have never heard about Aiflow, it’s a Python-based wor...
New
mhanberg
Hi everyone! The first release candidate for the Expert language server project is now available! We’ve published a press release detai...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews