Is there a way to access pyproject.toml?

Elixir Framework Forums Livebook Forum
,
1

Hi

Does somebody know if there’s a way to access the special cell pyproject.toml that is created when initializing Python in a Livebook from an Elixir cell? I’d just need the blob as a string.

2 Likes
2

It doesn’t seem there is a straight-forward location to fetch it. What is the use case?

1 Like
3

The use case is to reuse it to initialize Python in a FLAME runner. But I guess I can just not use that cell and create a module/function for initializing Python and use that on both sides (in the livebook and in the FLAME runner)

4

Ah, that’s something we want to explore. Honestly, I would instead try to find the folder where the Python packages are and transfer them as is instead. No need to download it again.

5

Tried. Failed… :slight_smile:
There might actually be a small bug in FLAME’s copy_paths option.

6

I manage to copy all the things over to the runner but somehow it still says Python is not initialized… might dig a bit more…

7

Alright, I managed to sync the entire pythonx cache directory by passing the following option to the FLAME pool:

copy_paths: Path.wildcard(Path.join(File.cwd!(), ".cache/pythonx/**"), match_dot: true)

However, I still need to run Pythonx.uv_init(my_toml) in order to initialize the NIF? But with the .venv in place, it won’t download stuff anymore. I still need that toml though, but that’s only because there’s currently no function to get the current project path inside .cache/pythonx. If I knew that, I could also only sync that project and didn’t have to sync the entire cache folder with all the projects in it. Plus, I could probably derife the necessary arguments to call Pythonx.NIF.init().

So I guess all in all the Pythonx module API could be improved for this use case but technically it’s not far from supporting it…?

8

Yes, it is not something we support yet but we want to make Pythonx work out of the box with FLAME, so any progress or PRs you can send along those lines will be welcome.

For example, we may be fine with moving the TOML to the .cache/pythonx, at least you can try it in a fork and see if that unblocks you.

2 Likes
9

Happy to support once we have an approach.

For example, we may be fine with moving the TOML to the .cache/pythonx

It’s a bit of a chicken/egg problem if I understand correctly. So when running in Livebook, the pyproject TOML is actually already inside .cache/pythonx. Concretely it’s in .cache/pythonx/PYTHONY_VERSION/uv/UV_VERSION/projects/MD5_OF_TOML/pyproject.toml. But without the toml it’s impossible to calculate that MD5_OF_TOML and derive the directory.

10

Perhaps uv_init should return the path to said project, and then we can keep it internally in Livebook or expose it as a var?

11

Yes that would probably work.

12

Or maybe it should return everything needed to call Pythonx.init/4 directly. Maybe a new struct with [:python_dl_path, :python_home_path, :python_executable_path, :sys_paths] that could be passed to sth like Pythonx.init/1?

13

It could be! Play with what is necessary and we can nitpick later!