Venomous - Erlport wrapper for managing concurrent python processes with ease.

:snake: :test_tube: Venomous aims to simplify the concurrent use of erlport Python Ports, focusing on dynamic extensibility such as spawning, reusing, and terminating processes on demand. It also handles unused processes, by killing them once they pass their configured inactive TTL. Venomous core functions ensure that whenever :boom: :EXIT signal appears, the Python process dies without further execution by killing :axe: its OS process (brutally).

This is my first attempt at creating an Elixir library. The idea stemmed from the challenge of properly exiting Python processes. Even after closing the Python port, execution would persist until the end of a function or iteration. My goal is to handle these exits effectively while also enabling process reuse, thus avoiding the constant spawning and stopping of new ones.

Any feedback would be greatly appreciated :smiling_face:

https://hexdocs.pm/venomous/Venomous.html

13 Likes

I’ve released 0.3.0 version of Venomous :snake:

  • fixed many major issues with how python processes were handled which caused zombie processes and weird behaviour.
  • multiple tasks ran on a single SnakeWorker will now stack instead of timing out
  • add option keywords

Changelog: Release v0.3.0 · RustySnek/Venomous · GitHub

2 Likes

Congrats on your first Elixir library. I like the delicious irony too that pythons aren’t venomous :slight_smile:

5 Likes

They must have drank some kind of toxic MIXture :wink:

3 Likes

thanks so much! I am using NimblePool to wrap python process running models. Your implementation to manage python process is way more sophisticated than mine so I guess I’m going to use it :heartpulse:

3 Likes

Thanks for providing that library.
However, I am unsure how to use it.
The documentation shows how to call a single python function.
Would you please explain how the python interpreter is started. How do I import some python modules and declare the function I want to call?

1 Like

Hey so basically as of now, all of the python modules (files) are loaded from PYTHONPATH env variable. So if you put your python modules inside python/ directory you would have to add that directory to PYTHONPATH envvar.

There is also a config for Venomous python processes to load the type encoder/decoder of erlport. I made a quick guide to this here: Quick guide on erlport Python API — Venomous v0.3.0

As for calling mutiple python instances you have to do it yourself for example.

 args = SnakeArgs.from_params(:time, :sleep, [0.1])
    1..100
    |> Enum.map(fn _ ->
      Task.async(fn -> python!(args) end)
    end)
    |> Task.await_many(:infinity)

Here Venomous will spawn as many processes as its allowed to via max_children configuration of SnakeSupervisor up to a 100. If it can’t spawn anymore it will just wait and reuse the already spawned ones once they are done with their tasks.

Feel free to ask if you need any more help. :smiling_face:

Thanks for the explanation. I guess I understand it now.
Perhaps it would be nice to put the python code “time.sleep(0.1)” also somewhere into the documentation. And also add the needed usage of PYTHONPATH when there is a need to call an own python module (which will be the case most of the time)

Yeah I’ll probably extend documentation and add few examples in the next release

2 Likes

I’ve released 0.4.0 version of Venomous :test_tube: :snake:

  • Included support for erlport python options. ex. module_paths, python_executable, packet_bytes…
  • Add named processes, separate from the regular SnakeManager ones
  • Fixed issue with lib breaking whenever python process was killed on exception…
  • Quicker exits whenver processes are spammed
  • Include examples in docs

Changelog: Release v0.4.0 · RustySnek/Venomous · GitHub

2 Likes