Hello,
I’d like to do some abominations on a running node with livebook. Currently since the node was built as a release, there is no mix support, so I cannot install any packages there.
Is it possible to add mix to the release? And if that is possible what are the implications of the Mix.install in a long-running system?
You can’t use packages beyond those available in the release. If you need notebook-specific dependencies, it is recommended to use the default Elixir standalone runtime with Mix.install and connect to the application node manually using Node.set_cookie/2 and Node.connect/1. Then you can use :erpc to run things on the application node. This way you can, for example, fetch data from the running system using :erpc.call and visualise it in the notebook with kino packages.
Thanks for the response!
Indeed what you are suggesting makes perfect sense, operate with data in the local context! The only downside of course being the fact that you lose all the advantages of running local code.
Do you think it would be possible to add a feature for switching running nodes seamlessly from a livebook in the future? I know that livebook has some kind of distribution, however I never dived into it, since I never was that interested in ML.
The only downside of course being the fact that you lose all the advantages of running local code.
You can still call your application code using :erpc.
Do you think it would be possible to add a feature for switching running nodes seamlessly from a livebook in the future?
What do you mean? The whole notebook runs in a single node, otherwise it would be harder to reproduce/configure, not to mention sending all variables across nodes.
Would it be possible in theory to abstract the node that a livebook runs on? Abstract not in the fact that it would run in ethereal, but the fact that you would be able to switch nodes seamlessly from the same livebook, I think this would have some interesting implications in infrastructure as code use-cases.
Would it be possible in theory to abstract the node that a livebook runs on?
That’s determined by the Runtime. You can use Attached to hook into any node or the default Elixir standalone, which always starts a new node under the hood. But they have very different use cases, with one significant different being exactly Mix.install/2. But as I mentioned the notebook runs in only one runtime at a time, because all the variables you define, etc is kept in that node.
Yes I am very aware of how livebook operates currently, the question is related to a possible future feature? I ask about the possibility of some DSL/feature for switching between nodes, where monaco editor that is used by livebook would be able to identify correctly autocomplete for functions and modules from different nodes and of course possibility of executing code from different nodes from the same livebook by just using some commands/code.
If I get the idea right, executing parts of a single notebook across multiple nodes is not something we plan to do.
Yes this is understandable, do you think that from elixir/OTP side there are any blockers currently?
Conceptually I feel it would impose a lot of complexity (also on the user) and be useful only in a handful of cases. On the technical side, the main issue is related to transferring evaluation state. Imagine you evaluate a cell in one node, and then you want to evaluate the next cell in the other node, we need to send all the binding between nodes, which is costly for large data. Even more so, when sending terms even across processes structural sharing is gone, so this would have heavy impact on memory usage, and operating on larger data would become inviable.
© Copyright Elixir Forum | Terms | Privacy & Cookies
