How do I redifine modules in livebook?

vgrechin · June 11, 2023, 8:00pm

I redefine a module in my livebook for educational purpose in order to demonstrate evolution of code within the same page.
I migrating the notebook to v0.9.2 and I encounter an error. How can I now redefine the module within the same page of Livebook?

** (CompileError) notebook/MyNotebooks/Notebook1.livemd#cell:xb7luhdi543e3ttejayalzbnuf3crtgd:1: module MyModule is already defined

benwilson512 · June 11, 2023, 8:08pm

A work around could be to do like:

defmodule Foo0 do
end

alias Foo0, as: Foo

And then later define a Foo1 and alias Foo1 as Foo.

Then all your code would of course be calling Foo.

jonatanklosko · June 11, 2023, 8:23pm

@vgrechin yeah, this is expected. We now do a more accurate dependency tracking between cells and module with the given name can be defined only in a single cell.

@benwilson512 very neat idea with the alias : )

Adzz · June 11, 2023, 8:27pm

I ran into this problem too. It makes it a nightmare to write tutorials and docs honestly because you can’t incrementally add things to the a module, you have to keep renaming it. It’s also different from how IEX works which is surprising…

vgrechin · June 11, 2023, 9:00pm

Thank you very much @benwilson512
you method works to some extend, but unfortunately page that I’m working on is effectively a chapter 2 of
Chris McCord’s ‘Metaprogramming Elixir’, which makes no use of the approach

benwilson512 · June 12, 2023, 12:45am

Ah yeah you’d need to unimport the previous ones if you’re importing them. import Assertion1, only: []

Definitely gets a bit ugly at that point.

shortlyportly · June 12, 2023, 10:04pm

Hi @jonatanklosko.

I have the same issue with livebook (which I think is an amazing piece of software). Would the core team be open to adding a feature that did allow you to redefine a module. Would we need to raise this against the GitHub repo?

cheers

Dave

josevalim · June 12, 2023, 11:37pm

We don’t plan to allow this because it completely messes up with Livebook ability of writing reproducible code.

If you redefine a module, and evaluate a previous cell, it gets the result of the new module, and now all of your computations may get out of sync.

shortlyportly · June 13, 2023, 5:30am

Thanks @josevalim. I understand. Livebook is still brilliant and I’ll carry on using it (even if I can’t redefine modules )

cheers

Dave

linusdm · June 13, 2023, 6:39am

I’m not sure if reproducibility can’t live together with redefining modules. For tutorials where there is a need to redefine a module over and over to evolve it over time, the only requirement is that the current cell, and all subsequent cells see the new module definition. I don’t think there would be confusion when the new module definition has no effect on previous cells, on the contrary, it’s kind of expected (which is exactly the reproducibility effect at work). If redefining a module has no effect on previous cells, then I don’t see how reproducibility (a non-negotiable property, I agree!) is in danger.
I see how it’s difficult to implement though, as module definitions live in a mutable global space.

You could think of the code cells as each defining their own closure, where the next code cell is closing over the previous cells to encompass it’s module definitions (and other bindings).

vgrechin · June 13, 2023, 7:13am

Dave, I can not confirm yet that this problem has any relation to Elixir as language. I’m only upgrading livebook which also a great tool for sharing thoughts under very active development.
My latest observation that these cells were fine on Elixir v1.13.4 + Livebook v0.6.2

Now I’ve got Elixir v1.14.2 and Livebook v0.9.2. I’ll conduct more trials when Elixir v15 is released in order to upgrade my environment.

linusdm · June 13, 2023, 7:31am

Just to be clear: a failure when redefining a module is intended behaviour since Livebook 0.8.0 (see the changelog about “Improved reproducibility of module definitions”). That’s what you’re seeing. You previously were running on an older version of Livebook that was ok with redefining modules (with the problem of not being fully reproducible, as José mentioned) which is why you had no problems back then. Nothing in Elixir itself changed in any way, regarding this. Livebook is more restrictive to make sure all livebooks are reproducible (run in the same way, no matter in which order you execute the cells). Regular Elixir allows you to redefine modules (it emits a warning if you do) afaik.

josevalim · June 13, 2023, 7:44am

That’s exactly the issue. Module names are global, so how can you make only subsequent cells see a given module definition? Even if you say: “well, we will purge and swap module versions before execution”, you can still have long-running processes and other functionality that will now be swapping modules on the fly.

Redefining a module is global mutable state and, if you want reproducible notebooks, then it won’t work.

linusdm · June 13, 2023, 7:59am

Ah yes, now I see where my thinking was not correct. I totally forgot about the concurrent aspects of running Elixir code .
There would indeed be no way to make such a system behave consistent. Thanks for clarifying!

vgrechin · June 13, 2023, 9:53pm

Thank you very, @linusdm for the detailed explanation of Livebook internals, but still can not observe evidence of the regression occurred in 0.8.0. When I tried to build 0.7.2 with Elixir 1.15-rc.2 just encountered a severe run-time error:

00:44:12.714 [error] #PID<0.521.0> running LivebookWeb.Endpoint (connection #PID<0.520.0>, stream id 1) terminated
Server: livebook.sleipner.ru:80 (http)
Request: GET /
** (exit) an exception was raised:
    ** (FunctionClauseError) no function clause matching in Livebook.Hubs.to_struct/1
        (livebook 0.7.2) lib/livebook/hubs.ex:104: Livebook.Hubs.to_struct(%{hub_emoji: "🏠", hub_name: "My Hub", id: "personal-hub", secret_key: "4oB-wLUGbX8SbVcfb6gzG4q4-8wSmmPvXOXmS7y2_wdMiyvSy9IDPr-83n2L7Dh28J2V_uDljhFjVtQ_v46fdw"})
        (elixir 1.15.0-rc.2) lib/enum.ex:1693: Enum."-map/2-lists^map/1-1-"/2
        (livebook 0.7.2) lib/livebook/hubs.ex:24: Livebook.Hubs.fetch_metadatas/0
        (livebook 0.7.2) lib/livebook_web/live/hooks/sidebar_hook.ex:12: LivebookWeb.SidebarHook.on_mount/4
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:148: anonymous fn/4 in Phoenix.LiveView.Lifecycle.mount/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:183: Phoenix.LiveView.Lifecycle.reduce_socket/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/utils.ex:319: anonymous fn/6 in Phoenix.LiveView.Utils.maybe_call_live_view_mount!/5
        (telemetry 1.2.1) /tmp/mix-local-installer-fetcher-U5e5yQ/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3
00:44:17.180 [error] #PID<0.527.0> running LivebookWeb.Endpoint (connection #PID<0.526.0>, stream id 1) terminated
Server: livebook.sleipner.ru:80 (http)
Request: GET /
** (exit) an exception was raised:
    ** (FunctionClauseError) no function clause matching in Livebook.Hubs.to_struct/1
        (livebook 0.7.2) lib/livebook/hubs.ex:104: Livebook.Hubs.to_struct(%{hub_emoji: "🏠", hub_name: "My Hub", id: "personal-hub", secret_key: "4oB-wLUGbX8SbVcfb6gzG4q4-8wSmmPvXOXmS7y2_wdMiyvSy9IDPr-83n2L7Dh28J2V_uDljhFjVtQ_v46fdw"})
        (elixir 1.15.0-rc.2) lib/enum.ex:1693: Enum."-map/2-lists^map/1-1-"/2
        (livebook 0.7.2) lib/livebook/hubs.ex:24: Livebook.Hubs.fetch_metadatas/0
        (livebook 0.7.2) lib/livebook_web/live/hooks/sidebar_hook.ex:12: LivebookWeb.SidebarHook.on_mount/4
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:148: anonymous fn/4 in Phoenix.LiveView.Lifecycle.mount/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:183: Phoenix.LiveView.Lifecycle.reduce_socket/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/utils.ex:319: anonymous fn/6 in Phoenix.LiveView.Utils.maybe_call_live_view_mount!/5
        (telemetry 1.2.1) /tmp/mix-local-installer-fetcher-U5e5yQ/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3
00:44:19.074 [error] #PID<0.533.0> running LivebookWeb.Endpoint (connection #PID<0.532.0>, stream id 1) terminated
Server: livebook.sleipner.ru:80 (http)
Request: GET /
** (exit) an exception was raised:
    ** (FunctionClauseError) no function clause matching in Livebook.Hubs.to_struct/1
        (livebook 0.7.2) lib/livebook/hubs.ex:104: Livebook.Hubs.to_struct(%{hub_emoji: "🏠", hub_name: "My Hub", id: "personal-hub", secret_key: "4oB-wLUGbX8SbVcfb6gzG4q4-8wSmmPvXOXmS7y2_wdMiyvSy9IDPr-83n2L7Dh28J2V_uDljhFjVtQ_v46fdw"})
        (elixir 1.15.0-rc.2) lib/enum.ex:1693: Enum."-map/2-lists^map/1-1-"/2
        (livebook 0.7.2) lib/livebook/hubs.ex:24: Livebook.Hubs.fetch_metadatas/0
        (livebook 0.7.2) lib/livebook_web/live/hooks/sidebar_hook.ex:12: LivebookWeb.SidebarHook.on_mount/4
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:148: anonymous fn/4 in Phoenix.LiveView.Lifecycle.mount/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:183: Phoenix.LiveView.Lifecycle.reduce_socket/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/utils.ex:319: anonymous fn/6 in Phoenix.LiveView.Utils.maybe_call_live_view_mount!/5
        (telemetry 1.2.1) /tmp/mix-local-installer-fetcher-U5e5yQ/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3