Error Spawning Process on Another Node

Questions / Help
,
1

Hi,

I’ve created a simple cluster by opening 2 IEX sessions on two different machines, and I can see the the nodes are clustered:

iex(dev@127.0.0.1)11> Node.list [:this, :visible]
[:"suv@127.0.0.1", :"dev@127.0.0.1"]

When I try running a simple process on the other node, I get this error:

 iex(dev@127.0.0.1)7> caller = self()
 #PID<0.115.0>
 iex(dev@127.0.0.1)8> Node.spawn(:"suv@127.0.0.1", fn -> send(caller, {:response, 1+2}) end)
 #PID<14963.130.0>
 
 11:43:45.064 [error] Process #PID<14963.130.0> on node :"suv@127.0.0.1" raised an exception
 ** (BadFunctionError) function #Function<43.125776118/0 in :erl_eval.expr/6> is invalid, likely because it points to an old version of the code
     :erlang.apply/2
 iex(dev@127.0.0.1)9> nil

The error implies :erl_eval.expr/6 is missing, so my first guess was differing versions of Elixir and/or Erlang/OTP on each node, but both have elixir 1.15 and OPT 26 as far as I can tell:

Node A:

iex(dev@127.0.0.1)6> System.build_info 
%{
   version: "1.15.5",
   date: "2023-08-28T11:58:30Z",
   otp_release: "26",
   build: "1.15.5 (compiled with Erlang/OTP 26)",
   revision: "9fd97c4" 
}

Node B:

 iex(suv@127.0.0.1)15> System.build_info
 %{
   version: "1.15.8",
   date: "2024-11-12T06:54:38Z",
   otp_release: "26",
   build: "1.15.8 (compiled with Erlang/OTP 26)",
   revision: ""
 }

When I use tab completion on :erl_eval.expr in IEX there is no :erl_eval.expr/6. Any idea what I might be doing wrong?

2

:wave:

What happens if you try

Node.spawn(:"suv@127.0.0.1", Kernel, :send, [caller, {:response, 1+2}])

?

Re erl_eval in #Function<43.125776118/0 in :erl_eval.expr/6> I think it just means that the anonymous function was defined in IEx. It might or might not mean that erl_eval is the problem. It might as well be Kernel.send/1, if that module / function were updated between 1.15.5 and 1.15.8, which they probably were, at least in “version”.

3

That worked, thanks!

For my own curiosity, how could I verify where the issue might lie? Look at the Kernel.send source code for each of the Elixir versions or is there a better way?

4

I would guess that the closures capture module versions (MD5 by default) in addition to module and function names and since they are different in different versions of Elixir, it results in an error. And maybe sending Node.spawn(:"suv@127.0.0.1", Kernel, :send, [caller, {:response, 1+2}]) doesn’t concern itself with module versions and just runs apply(Kernel, :send, [caller, {:response, 1+2}]) so it works across Elixir versions. But it can possibly produce different results from if this was executed locally. But I guess that’s usually expected with distributed systems.

1 Like
5

I would guess that the closures capture module versions

erlang — erts v15.1.2 seems to support this guess:

When a local fun is called, the same version of the code that created the fun is called (even if a newer version of the module has been loaded).

That version seems to be stored in :new_uniq attribute. Maybe it does some rolling hash of the versions of all the modules mentioned in the function body. :person_shrugging:

iex> Kernel.module_info[:attributes][:vsn]
#==> [283001957550643237782290081599671487237]

iex> :binary.encode_unsigned 283001957550643237782290081599671487237
#==> <<212, 232, 49, 198, 154, 127, 2, 77, 55, 2, 94, 199, 16, 227, 203, 5>>

iex> caller = self()
iex> f = fn -> Kernel.send(caller, {:response, 1+2}) end
iex> Function.info(f, :new_uniq)
#==> {:new_uniq, <<74, 179, 14, 23, 76, 2, 152, 184, 122, 207, 206, 42, 63, 68, 21, 64>>}
6

When spawning a remote process like this, does the ‘work’ (1+2 in this case) get evaluated on the local node?

I’d like the ‘work’ to be evaluated on the remote node, and when I replace 1+2 with Node.self(), it appears that Node.self()is evaluated locally, and that the remote node is sending the already computed value back to the local node:

 iex(dev@127.0.0.1)7> Node.spawn(:"suv@127.0.0.1", Kernel, :send, [caller, {:response, Node.self}])
 #PID<13803.123.0>
 iex(dev@127.0.0.1)8> flush
{:response, :"dev@127.0.0.1}
:ok

I’m a relative Elixir noob, so not quite grasping how to make the remote node do the work :slight_smile:

Thanks for your help

7

When spawning a remote process like this, does the ‘work’ (1+2 in this case) get evaluated on the local node?

Arguments (it’s just a normal list) are “evaluated” on the calling node, the function is executed on the remote node.

It’s the same as:

iex(dev@127.0.0.1)> caller = self()
iex(dev@127.0.0.1)> args = [caller, {:response, Node.self}]
#==> [#PID<0.114.0>, {:response, :"dev@127.0.0.1"}]

iex(dev@127.0.0.1)> Node.spawn(:"suv@127.0.0.1", Kernel, :send, args)
iex(dev@127.0.0.1)> flush()
#==> {:response, :"dev@127.0.0.1)"}

You can make the called function process its args and then that part would be handled by the remote node too:

iex(suv@127.0.0.1)> defmodule Worker do
                      def do_work, do: {:response, Node.self()}
                    end

iex(suv@127.0.0.1)> Worker.do_work()
#==> {:response, :"suv@127.0.0.1)"}

# -------------------- back to dev@127.0.0.1 -------------------------

iex(dev@127.0.0.1)> :erpc.call(:"suv@127.0.0.1", Worker, :do_work, [])
#==> {:response, :"suv@127.0.0.1)"}

:erpc.call here just executes the function remotely and sends back the results, similar to your Node.spawn example with send. I just wanted to mention it as it’s a pretty useful module.

2 Likes
8

Cool, that’s what I figured. So you either need code on the remote node which you can invoke from the local node, or pass a function to the remote node (assuming both nodes have compatible versions).

Thanks for the tip on :erpc.call, that looks pretty useful :slight_smile: