Venomous aims to simplify the concurrent use of erlport Python Ports, focusing on dynamic extensibility such as spawning, reusing, and terminating processes on demand. It also handles unused processes, by killing them once they pass their configured inactive TTL. Venomous core functions ensure that whenever :EXIT signal appears, the Python process dies without further execution by killing its OS process (brutally).
This is my first attempt at creating an Elixir library. The idea stemmed from the challenge of properly exiting Python processes. Even after closing the Python port, execution would persist until the end of a function or iteration. My goal is to handle these exits effectively while also enabling process reuse, thus avoiding the constant spawning and stopping of new ones.
thanks so much! I am using NimblePool to wrap python process running models. Your implementation to manage python process is way more sophisticated than mine so I guess I’m going to use it
Thanks for providing that library.
However, I am unsure how to use it.
The documentation shows how to call a single python function.
Would you please explain how the python interpreter is started. How do I import some python modules and declare the function I want to call?
Hey so basically as of now, all of the python modules (files) are loaded from PYTHONPATH env variable. So if you put your python modules inside python/ directory you would have to add that directory to PYTHONPATH envvar.
Here Venomous will spawn as many processes as its allowed to via max_children configuration of SnakeSupervisor up to a 100. If it can’t spawn anymore it will just wait and reuse the already spawned ones once they are done with their tasks.
Thanks for the explanation. I guess I understand it now.
Perhaps it would be nice to put the python code “time.sleep(0.1)” also somewhere into the documentation. And also add the needed usage of PYTHONPATH when there is a need to call an own python module (which will be the case most of the time)
I have built an ETL in Python that I want to call from Elixir. I wanted to ask if you have any suggestions on how to best pass maps/dictionaries between Elixir and Python. Some of the maps/dicts will contain simple structures, but others will contain more complex structures, such as pandas DataFrames.
I have looked into both serialization via JSON and writing custom functions on each side. I assume this will be quite a common use case for Venomous, so I wanted to ask your opinion on this.
Thanks for a great library with good documentation; it’s been a great introduction to the world of Elixir!
Hey, for simple classes that can be easily serialized with .__dict__ you can just handle that recursively for basic data types. venomous.py provides a function that does handles such cases and encodes all strings into ‘utf-8’ so they won’t appear as charlists on elixir’s side.
def encode_basic_type_strings(data: Any):
"""
encodes str into utf-8 bytes
handles VenomousTrait classes into structs
converts non VenomousTrait classes into .__dict__
"""
if isinstance(data, str):
return data.encode("utf-8")
elif isinstance(data, (list, tuple, set)):
return type(data)(encode_basic_type_strings(item) for item in data)
elif isinstance(data, dict):
return {
encode_basic_type_strings(key): encode_basic_type_strings(value)
for key, value in data.items()
}
elif isinstance(data, VenomousTrait):
return data.into_erl()
elif (_dic := getattr(data, "__dict__", None)) != None:
return encode_basic_type_strings(_dic)
else:
return data
If you want to maintain the structs/classes between elixir/python you can experiment with VenomousTrait class all tho I haven’t documented it very well yet.
As for the more complex structures you have to handle them individually, like for example DataFrames provides to_dict() function which returns a clean dict with data. All of the logic of conversion should be put inside the encoder/decoder functions of erlport. So for the DataFrame you could do:
```python
# encoder.py
from typing import Any
from erlport.erlang import set_decoder, set_encoder
from erlport.erlterms import Atom
from pandas import DataFrame
from venomous import decode_basic_types_strings, encode_basic_type_strings
def handle_types():
set_encoder(encoder)
set_decoder(decoder)
return Atom("ok".encode("utf-8"))
def encoder(value: Any):
if isinstance(value, DataFrame):
return encode_basic_type_strings(value.to_dict())
return encode_basic_type_strings(value)
def decoder(value: Any):
return decode_basic_types_strings(value)
I’m building a PoC on some machine learning API which uses Elixir to manage Python processes for NLP task.
There is a small cold start when the method is invoke the first time with Venoumous.python call. I wonder if there is an easy way to pre-start some worker so there is no cold start time when running the program?
I’m looking at the SnakeWorker/Supervisor but not sure if it’s the correct place.
I have added Venomous.preload_snakes/1 in the 0.7.5 release, which basically starts x amount of processes with :ready state. So you can basically start workers at the start of your program with:
:ok = Venomous.preload_snakes(10) # Starts 10 workers
{:retrieve_error, :max_children} = Venomous.preload_snakes(-1) # Starts all available workers
Hey, I don’t encounter such problem when I do Application.stop(:venomous). However you mentioned that you exit a different supervisor so perhaps you would have to link them so they terminate alongside each other? Calling stop on :venomous is also a way.
Actually it took a while for the process to be removed. After I waited a bit, ps -aux | grep erlport does not show running worker anymore so all good!
I’ve been using the preload as well and it works perfectly. A little curious about the reason why you make the return value when using -1 as {:retrieve_error, :max_children} instead of :ok when it’s successful?
It wasn’t really well thought out as if you just supply the function with -1 it will keep on spawning workers until it encounters the error which in this case will be the :max_children. It’s kind of a way of signaling that you have reached the limit. I might change it later on to make a little bit more sense as it’s not really an error if everything did work as intended.
Venomous aims to simplify the concurrent use of erlport Python Ports, focusing on dynamic extensibility such as spawning, reusing, and terminating processes on demand. It also handles unused processes, by killing them once they pass their configured inactive TTL. Venomous core functions ensure that whenever 
its OS process (brutally).
Hot reloading for python modules.
