How do I redifine modules in livebook?

New joiner here. It’s very unintuitive. Wouldn’t a solution be to restart a livebook (a bit like jupyter restarts a kernel wiping all state)?

(I am on mac (v.0.14.5) but could not find a way to restart.)

Thank you very, @linusdm for the detailed explanation of Livebook internals, but still can not observe evidence of the regression occurred in 0.8.0. When I tried to build 0.7.2 with Elixir 1.15-rc.2 just encountered a severe run-time error:

00:44:12.714 [error] #PID<0.521.0> running LivebookWeb.Endpoint (connection #PID<0.520.0>, stream id 1) terminated
Server: livebook.sleipner.ru:80 (http)
Request: GET /
** (exit) an exception was raised:
    ** (FunctionClauseError) no function clause matching in Livebook.Hubs.to_struct/1
        (livebook 0.7.2) lib/livebook/hubs.ex:104: Livebook.Hubs.to_struct(%{hub_emoji: "🏠", hub_name: "My Hub", id: "personal-hub", secret_key: "4oB-wLUGbX8SbVcfb6gzG4q4-8wSmmPvXOXmS7y2_wdMiyvSy9IDPr-83n2L7Dh28J2V_uDljhFjVtQ_v46fdw"})
        (elixir 1.15.0-rc.2) lib/enum.ex:1693: Enum."-map/2-lists^map/1-1-"/2
        (livebook 0.7.2) lib/livebook/hubs.ex:24: Livebook.Hubs.fetch_metadatas/0
        (livebook 0.7.2) lib/livebook_web/live/hooks/sidebar_hook.ex:12: LivebookWeb.SidebarHook.on_mount/4
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:148: anonymous fn/4 in Phoenix.LiveView.Lifecycle.mount/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:183: Phoenix.LiveView.Lifecycle.reduce_socket/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/utils.ex:319: anonymous fn/6 in Phoenix.LiveView.Utils.maybe_call_live_view_mount!/5
        (telemetry 1.2.1) /tmp/mix-local-installer-fetcher-U5e5yQ/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3
00:44:17.180 [error] #PID<0.527.0> running LivebookWeb.Endpoint (connection #PID<0.526.0>, stream id 1) terminated
Server: livebook.sleipner.ru:80 (http)
Request: GET /
** (exit) an exception was raised:
    ** (FunctionClauseError) no function clause matching in Livebook.Hubs.to_struct/1
        (livebook 0.7.2) lib/livebook/hubs.ex:104: Livebook.Hubs.to_struct(%{hub_emoji: "🏠", hub_name: "My Hub", id: "personal-hub", secret_key: "4oB-wLUGbX8SbVcfb6gzG4q4-8wSmmPvXOXmS7y2_wdMiyvSy9IDPr-83n2L7Dh28J2V_uDljhFjVtQ_v46fdw"})
        (elixir 1.15.0-rc.2) lib/enum.ex:1693: Enum."-map/2-lists^map/1-1-"/2
        (livebook 0.7.2) lib/livebook/hubs.ex:24: Livebook.Hubs.fetch_metadatas/0
        (livebook 0.7.2) lib/livebook_web/live/hooks/sidebar_hook.ex:12: LivebookWeb.SidebarHook.on_mount/4
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:148: anonymous fn/4 in Phoenix.LiveView.Lifecycle.mount/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:183: Phoenix.LiveView.Lifecycle.reduce_socket/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/utils.ex:319: anonymous fn/6 in Phoenix.LiveView.Utils.maybe_call_live_view_mount!/5
        (telemetry 1.2.1) /tmp/mix-local-installer-fetcher-U5e5yQ/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3
00:44:19.074 [error] #PID<0.533.0> running LivebookWeb.Endpoint (connection #PID<0.532.0>, stream id 1) terminated
Server: livebook.sleipner.ru:80 (http)
Request: GET /
** (exit) an exception was raised:
    ** (FunctionClauseError) no function clause matching in Livebook.Hubs.to_struct/1
        (livebook 0.7.2) lib/livebook/hubs.ex:104: Livebook.Hubs.to_struct(%{hub_emoji: "🏠", hub_name: "My Hub", id: "personal-hub", secret_key: "4oB-wLUGbX8SbVcfb6gzG4q4-8wSmmPvXOXmS7y2_wdMiyvSy9IDPr-83n2L7Dh28J2V_uDljhFjVtQ_v46fdw"})
        (elixir 1.15.0-rc.2) lib/enum.ex:1693: Enum."-map/2-lists^map/1-1-"/2
        (livebook 0.7.2) lib/livebook/hubs.ex:24: Livebook.Hubs.fetch_metadatas/0
        (livebook 0.7.2) lib/livebook_web/live/hooks/sidebar_hook.ex:12: LivebookWeb.SidebarHook.on_mount/4
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:148: anonymous fn/4 in Phoenix.LiveView.Lifecycle.mount/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/lifecycle.ex:183: Phoenix.LiveView.Lifecycle.reduce_socket/3
        (phoenix_live_view 0.18.2) lib/phoenix_live_view/utils.ex:319: anonymous fn/6 in Phoenix.LiveView.Utils.maybe_call_live_view_mount!/5
        (telemetry 1.2.1) /tmp/mix-local-installer-fetcher-U5e5yQ/deps/telemetry/src/telemetry.erl:321: :telemetry.span/3

Ah yes, now I see where my thinking was not correct. I totally forgot about the concurrent aspects of running Elixir code .
There would indeed be no way to make such a system behave consistent. Thanks for clarifying!

That’s exactly the issue. Module names are global, so how can you make only subsequent cells see a given module definition? Even if you say: “well, we will purge and swap module versions before execution”, you can still have long-running processes and other functionality that will now be swapping modules on the fly.

Redefining a module is global mutable state and, if you want reproducible notebooks, then it won’t work.

Just to be clear: a failure when redefining a module is intended behaviour since Livebook 0.8.0 (see the changelog about “Improved reproducibility of module definitions”). That’s what you’re seeing. You previously were running on an older version of Livebook that was ok with redefining modules (with the problem of not being fully reproducible, as José mentioned) which is why you had no problems back then. Nothing in Elixir itself changed in any way, regarding this. Livebook is more restrictive to make sure all livebooks are reproducible (run in the same way, no matter in which order you execute the cells). Regular Elixir allows you to redefine modules (it emits a warning if you do) afaik.

Dave, I can not confirm yet that this problem has any relation to Elixir as language. I’m only upgrading livebook which also a great tool for sharing thoughts under very active development.
My latest observation that these cells were fine on Elixir v1.13.4 + Livebook v0.6.2

Now I’ve got Elixir v1.14.2 and Livebook v0.9.2. I’ll conduct more trials when Elixir v15 is released in order to upgrade my environment.

I’m not sure if reproducibility can’t live together with redefining modules. For tutorials where there is a need to redefine a module over and over to evolve it over time, the only requirement is that the current cell, and all subsequent cells see the new module definition. I don’t think there would be confusion when the new module definition has no effect on previous cells, on the contrary, it’s kind of expected (which is exactly the reproducibility effect at work). If redefining a module has no effect on previous cells, then I don’t see how reproducibility (a non-negotiable property, I agree!) is in danger.
I see how it’s difficult to implement though, as module definitions live in a mutable global space.

You could think of the code cells as each defining their own closure, where the next code cell is closing over the previous cells to encompass it’s module definitions (and other bindings).

Thanks @josevalim. I understand. Livebook is still brilliant and I’ll carry on using it (even if I can’t redefine modules )

cheers

Dave

We don’t plan to allow this because it completely messes up with Livebook ability of writing reproducible code.

If you redefine a module, and evaluate a previous cell, it gets the result of the new module, and now all of your computations may get out of sync.

Hi @jonatanklosko.

I have the same issue with livebook (which I think is an amazing piece of software). Would the core team be open to adding a feature that did allow you to redefine a module. Would we need to raise this against the GitHub repo?

cheers

Dave