Nx.from_binary/2 raises (RuntimeError) Unable to get dimensions above specific binary size

Elixir Framework Forums Nx Forum
,
1

I’m calling Nx.from_binary(binary, :u8)

If the binary_size is above 2147483647 (which is 2^31 - 1) then it raises the below error:

iex(dev@dev)49> Nx.from_binary(binary_part(bin, 0, 2147483647), :u8)
#Nx.Tensor<
  u8[2147483647]
  EXLA.Backend<host:0, 0.4210033359.2202140741.62473>
  [0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, ...]
>
iex(dev@dev)50> Nx.from_binary(binary_part(bin, 0, 2147483648), :u8)
** (RuntimeError) Unable to get dimensions.
    (exla 0.7.2) lib/exla/shape.ex:89: EXLA.Shape.unwrap!/1
    (exla 0.7.2) lib/exla/shape.ex:29: EXLA.Shape.make_shape/2
    (exla 0.7.2) lib/exla/backend.ex:45: EXLA.Backend.from_binary/3
    iex:50: (file)

Is this intended behavior or fundamental limitation? I don’t find it documented in EXLA or NX.

2

This is probably a bug. Please try out main branch with {:exla, github: "elixir-nx/nx", sparse: "exla"}, and if the problem persists, open an issue so that I can tackle it.

We’ve just merged a new PR yesterday that changes shape handling, so this might have gone away.

3

Actually, I might have found the bug :slight_smile:
Please try the branch: pv-fix/exla-get-tuple

2 Likes
4

This branch definitely fixes it, tried with a huge binary.
I hope it will be merged soon, until then I will use it from GitHub. Thanks for the fix!

5

Awesome! I just have to think about how to test this and it’ll be merged