I have built an ETL in Python that I want to call from Elixir. I wanted to ask if you have any suggestions on how to best pass maps/dictionaries between Elixir and Python. Some of the maps/dicts will contain simple structures, but others will contain more complex structures, such as pandas DataFrames.
I have looked into both serialization via JSON and writing custom functions on each side. I assume this will be quite a common use case for Venomous, so I wanted to ask your opinion on this.
Thanks for a great library with good documentation; it’s been a great introduction to the world of Elixir!
Hey, for simple classes that can be easily serialized with .__dict__ you can just handle that recursively for basic data types. venomous.py provides a function that does handles such cases and encodes all strings into ‘utf-8’ so they won’t appear as charlists on elixir’s side.
def encode_basic_type_strings(data: Any):
"""
encodes str into utf-8 bytes
handles VenomousTrait classes into structs
converts non VenomousTrait classes into .__dict__
"""
if isinstance(data, str):
return data.encode("utf-8")
elif isinstance(data, (list, tuple, set)):
return type(data)(encode_basic_type_strings(item) for item in data)
elif isinstance(data, dict):
return {
encode_basic_type_strings(key): encode_basic_type_strings(value)
for key, value in data.items()
}
elif isinstance(data, VenomousTrait):
return data.into_erl()
elif (_dic := getattr(data, "__dict__", None)) != None:
return encode_basic_type_strings(_dic)
else:
return data
If you want to maintain the structs/classes between elixir/python you can experiment with VenomousTrait class all tho I haven’t documented it very well yet.
As for the more complex structures you have to handle them individually, like for example DataFrames provides to_dict() function which returns a clean dict with data. All of the logic of conversion should be put inside the encoder/decoder functions of erlport. So for the DataFrame you could do:
```python
# encoder.py
from typing import Any
from erlport.erlang import set_decoder, set_encoder
from erlport.erlterms import Atom
from pandas import DataFrame
from venomous import decode_basic_types_strings, encode_basic_type_strings
def handle_types():
set_encoder(encoder)
set_decoder(decoder)
return Atom("ok".encode("utf-8"))
def encoder(value: Any):
if isinstance(value, DataFrame):
return encode_basic_type_strings(value.to_dict())
return encode_basic_type_strings(value)
def decoder(value: Any):
return decode_basic_types_strings(value)
I’m building a PoC on some machine learning API which uses Elixir to manage Python processes for NLP task.
There is a small cold start when the method is invoke the first time with Venoumous.python call. I wonder if there is an easy way to pre-start some worker so there is no cold start time when running the program?
I’m looking at the SnakeWorker/Supervisor but not sure if it’s the correct place.
I have added Venomous.preload_snakes/1 in the 0.7.5 release, which basically starts x amount of processes with :ready state. So you can basically start workers at the start of your program with:
:ok = Venomous.preload_snakes(10) # Starts 10 workers
{:retrieve_error, :max_children} = Venomous.preload_snakes(-1) # Starts all available workers
Hey, I don’t encounter such problem when I do Application.stop(:venomous). However you mentioned that you exit a different supervisor so perhaps you would have to link them so they terminate alongside each other? Calling stop on :venomous is also a way.
Actually it took a while for the process to be removed. After I waited a bit, ps -aux | grep erlport does not show running worker anymore so all good!
I’ve been using the preload as well and it works perfectly. A little curious about the reason why you make the return value when using -1 as {:retrieve_error, :max_children} instead of :ok when it’s successful?
It wasn’t really well thought out as if you just supply the function with -1 it will keep on spawning workers until it encounters the error which in this case will be the :max_children. It’s kind of a way of signaling that you have reached the limit. I might change it later on to make a little bit more sense as it’s not really an error if everything did work as intended.
