Venomous - Erlport wrapper for managing concurrent python processes with ease

I’ve released 0.3.0 version of Venomous

• fixed many major issues with how python processes were handled which caused zombie processes and weird behaviour.
• multiple tasks ran on a single SnakeWorker will now stack instead of timing out
• add option keywords

Changelog: Release v0.3.0 · RustySnek/Venomous · GitHub

Congrats on your first Elixir library. I like the delicious irony too that pythons aren’t venomous

They must have drank some kind of toxic MIXture

thanks so much! I am using NimblePool to wrap python process running models. Your implementation to manage python process is way more sophisticated than mine so I guess I’m going to use it

Thanks for providing that library.
However, I am unsure how to use it.
The documentation shows how to call a single python function.
Would you please explain how the python interpreter is started. How do I import some python modules and declare the function I want to call?

Hey so basically as of now, all of the python modules (files) are loaded from PYTHONPATH env variable. So if you put your python modules inside python/ directory you would have to add that directory to PYTHONPATH envvar.

There is also a config for Venomous python processes to load the type encoder/decoder of erlport. I made a quick guide to this here: Quick guide on erlport Python API — Venomous v1.0.0

As for calling mutiple python instances you have to do it yourself for example.

 args = SnakeArgs.from_params(:time, :sleep, [0.1])
    1..100
    |> Enum.map(fn _ ->
      Task.async(fn -> python!(args) end)
    end)
    |> Task.await_many(:infinity)

Here Venomous will spawn as many processes as its allowed to via max_children configuration of SnakeSupervisor up to a 100. If it can’t spawn anymore it will just wait and reuse the already spawned ones once they are done with their tasks.

Feel free to ask if you need any more help.

Thanks for the explanation. I guess I understand it now.
Perhaps it would be nice to put the python code “time.sleep(0.1)” also somewhere into the documentation. And also add the needed usage of PYTHONPATH when there is a need to call an own python module (which will be the case most of the time)

Yeah I’ll probably extend documentation and add few examples in the next release

I’ve released 0.4.0 version of Venomous

• Included support for erlport python options. ex. module_paths, python_executable, packet_bytes..
• Add named processes, separate from the regular SnakeManager ones
• Fixed issue with lib breaking whenever python process was killed on exception…
• Quicker exits whenver processes are spammed
• Include examples in docs

Changelog: Release v0.4.0 · RustySnek/Venomous · GitHub

I’ve released 0.5.1 version of Venomous which adds optional Hot reloading for python modules.

To enable the hot reloading:

• Install python watchdog dependancy using mix venomous.watchdog install
• Enable serpent_watcher in your dev config:

  config :venomous, :serpent_watcher, enable: true
  

• Add your module paths in snake_manager config:

  config :venomous, :snake_manager, %{
    python_opts: [
      module_paths: ["my_python_modules/", ...]
    ]
  }
  

Now all modules inside the configured module_paths should reload on edit.