Python module not found with erlport in release mode

Questions / Help
1

Hello everyone,

I do have a strange error happening when using erlport and python.

** (ErlangError) Erlang error: {:python, :"builtins.ModuleNotFoundError", 'No module named \'cv2\'', [' File "/export/koko/koko/_build/prod/rel/koko/lib/face_recon2-0.1.0/priv/python_scripts/face_recon2.py", line 1, in <module>\n import cv2\n', ' File "/export/koko/koko/_build/prod/rel/koko/lib/erlport-0.10.1/priv/python3/erlport/erlang.py", line 225, in _incoming_call\n f = __import__(mod, {}, {}, [objects[0]])\n', ' File "/export/koko/koko/_build/prod/rel/koko/lib/erlport-0.10.1/priv/python3/erlport/erlang.py", line 233, in _call_with_error_handler\n function(*args)\n']}
** (EXIT) an exception was raised:
** (stop) exited in: GenServer.call(#PID<0.2634.0>, {:recon_faces, "/export/koko/uploads/post_processing/MTY0NzczMjU3NzY1Nw/screenshots"}, :infinity)

I do have cv2 installed with opencv-python, opencv-contrib-python, but it looks like it does not find it.

What is strange is… it’s working in dev, it’s working with MIX_ENV=prod mix phx.server, but not when starting the release.

Did I forgot to set some python environment variables?

I start the erlport process with…

  @impl true
  def init(_) do
    path = Path.join([:code.priv_dir(:face_recon2), "python_scripts"])

    with {:ok, pid} <- :python.start([{:python_path, to_charlist(path)}, {:python, 'python3'}]) do
      Logger.info("[#{__MODULE__}] Started python worker #{inspect self()}")
      {:ok, pid}
    end
  end
...
  @impl true
  def handle_call({:recon_faces, path}, _from, pid) do
    result = case :python.call(pid, :face_recon2, :recon_faces, [path, training_set()]) do
      {'ok', map_message} -> {:ok, sanitize(map_message)}
      {'error', error} -> {:error, to_string(error)}
      any -> {:error, "Unknown error #{inspect any}"}
    end
    {:reply, result, pid}
  end

I know it’s very specific, but if someone can help me understand why the python3 cv2 library is not found, I’ll be very happy :slight_smile:

Thanks for taking time

3 Likes
2

Did you try setting PYTHONPATH env variable?

If it is working in one config and not working in other - you can check search path for modules:

import sys
sys.path

If you can diff sys.path in release and dev, may be you can find answer of what is missing?

3 Likes
3

Thank You for your help, I will try and report.

Could You tell me more about this PYTHONPATH? For sure I did not use it :slight_smile:

2 Likes
4

Python searches for modules in following places:

  1. tries to find in current directory from where script is run or interpreter is run
  2. paths added using PYTHONPATH
  3. python runtime

When using virtual environments - source <venv>/bin/activate it adds <venv>/lib/python<version>/site-packages folder to the search path.

Additional paths can be added using PYTHONPATH. Erlport also injects PYTHONPATH if it is set.

I feel you will find answer in diff from sys.path in dev and release environments.

It is unable to resolve cv2 module - mostly it has something to do with module search path.

4 Likes
5

Thank You,

I have found the path… unfortunately, I cannot test it yet, because someone is working on the site.

But it looks this should solve the problem :slight_smile:

2 Likes
6

I updated the systemd script that start my application to include all path found with sys.path.

And now it works :slight_smile:

Thank You very much

2 Likes
7

You are welcome. Memories from the past - it was hell supporting python2 and python3 in production.

Just curious - you are using virtualenv in production ?

2 Likes
8

No I don’t, I use one vm per apps, with/without Python.

If I had to share multiple Python environment per vm, I would use virtualenv.

But I am using it on my development machine :slight_smile:

2 Likes
9

virtualenv is also used to isolate the app from changes in system dependencies and upgrades in per vm machines - to pin down to a python version and pip dependencies.

Things evolve rapidly - don’t how people are solving this. Luckily I am out of python deployments from a couple of years.

2 Likes
10

can you share your env file since I am facing the same issue but I don’t know what to do as in my sys path I do have python
:/Library/PostgreSQL/16/bin:/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/Current/bin

but everytime I run :python.start it simply throws an error :-
{:error, :python_not_found}

So I created a virtual environment and then activated and then tried and then it works so I am pretty sure its something related to path. I tried this also :-

System.put_env("ERLPORT_PYTHON", "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/Current/bin/python3")
:ok
iex(2)> :python.start()
{:ok, #PID<0.186.0>}
iex(3)>

Again I don’t want to set the path this manner

11

I would suggest using asdf or poetry for python management as they will handle the shims

1 Like