You can also lower the number of jobs using the —jobs= flag
seanmor5
Author of Genetic Algorithms in Elixir
karang
Thanks for the tip. I think now the problem is I am not setting the flags properly. I tried this:
$ export BAZEL_FLAGS=--discard_analysis_cache,--nokeep_state_after_build,--notrack_incremental_state,--jobs=8,--host_jvm_args=-Xmx8g
But it did not seem to make a difference. I tried some different combinations of the above all to no avail. All cores were running (I thought only 8 should be active) and it still ran out of memory. I also tried to set it like this:
$ BAZEL_FLAGS=--discard_analysis_cache,--nokeep_state_after_build,--notrack_incremental_state,--jobs=8,--host_jvm_args=-Xmx8g mix deps.compile exla
But that didn’t seem to work either. Is this the correct way to set the BAZEL_FLAGS variable. Thanks!
0
zeio
Is it possible to use a precompiled xla library? I’ve found this repo but somewhy still cannot run my models on gpu. When I am trying to explicitly specify EXLA compiler in the code, I get the following error:
01:01:51.192 [error] GenServer EXLA.Client terminating
** (RuntimeError) Could not find registered platform with name: "cuda". Available platform names are: Host
(exla 0.1.0-dev) lib/exla/client.ex:153: EXLA.Client.unwrap!/1
(exla 0.1.0-dev) lib/exla/client.ex:134: EXLA.Client.build_client/2
(exla 0.1.0-dev) lib/exla/client.ex:94: EXLA.Client.handle_call/3
(stdlib 3.16.1) gen_server.erl:721: :gen_server.try_handle_call/4
(stdlib 3.16.1) gen_server.erl:750: :gen_server.handle_msg/6
(stdlib 3.16.1) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
I’ve an exla dependency in my mix.exs:
{:exla, "~> 0.1.0-dev", github: "elixir-nx/nx", sparse: "exla"}
I’ve set up configuration in the config/config.exs:
config :nx, :default_defn_options, [compiler: EXLA, client: :cuda]
config :exla, :clients, cuda: [platform: :cuda], default: [platform: :cuda]
I’ve the following environment variables as well:
XLA_BUILD=true
XLA_TARGET=cuda111
EXLA_TARGET=cuda
TF_CUDA_VERSION='11.2'
And off course I’ve installed cuda and cudnn so it is recognized by the tensorflow library. Command python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))" produces following output:
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
Do I miss something?
0
josevalim
Creator of Elixir
Remove XLA_BUILD=true and it should pick up a precompiled version based on your XLA_TARGET. If it doesn’t work, please post the full results of mix deps.get plus mix compile. 
0
zeio
Thank you for the response! I’ve followed your instructions and reset the env variable XLA_BUILD. Then I deleted the deps and _build folders and run command mix deps.get, which produced the following output:
* Getting nx (https://github.com/zeionara/nx.git)
remote: Enumerating objects: 10503, done.
remote: Counting objects: 100% (2118/2118), done.
remote: Compressing objects: 100% (304/304), done.
remote: Total 10503 (delta 1908), reused 1924 (delta 1807), pack-reused 8385
Receiving objects: 100% (10503/10503), 2.68 MiB | 1.75 MiB/s, done.
Resolving deltas: 100% (6947/6947), done.
* Getting exla (https://github.com/elixir-nx/nx.git)
remote: Enumerating objects: 10586, done.
remote: Counting objects: 100% (2490/2490), done.
remote: Compressing objects: 100% (427/427), done.
remote: Total 10586 (delta 2159), reused 2270 (delta 2054), pack-reused 8096
Receiving objects: 100% (10586/10586), 2.85 MiB | 1.59 MiB/s, done.
Resolving deltas: 100% (7009/7009), done.
* Getting axon (https://github.com/zeionara/axon.git - origin/epoch-completion-handler)
remote: Enumerating objects: 2727, done.
remote: Counting objects: 100% (1903/1903), done.
remote: Compressing objects: 100% (1123/1123), done.
remote: Total 2727 (delta 1406), reused 1175 (delta 761), pack-reused 824
Receiving objects: 100% (2727/2727), 9.03 MiB | 1.23 MiB/s, done.
Resolving deltas: 100% (1864/1864), done.
* Getting axon_onnx (https://github.com/zeionara/axon_onnx.git - origin/master)
remote: Enumerating objects: 156, done.
remote: Counting objects: 100% (156/156), done.
remote: Compressing objects: 100% (109/109), done.
remote: Total 156 (delta 63), reused 127 (delta 40), pack-reused 0
Receiving objects: 100% (156/156), 54.87 KiB | 1021.00 KiB/s, done.
Resolving deltas: 100% (63/63), done.
Resolving Hex dependencies...
Dependency resolution completed:
Unchanged:
dialyxir 1.1.0
elixir_make 0.6.3
elixir_uuid 1.2.1
erlex 0.2.6
optimus 0.2.0
protox 1.4.0
table_rex 3.1.1
xla 0.2.0
* Getting dialyxir (Hex package)
* Getting optimus (Hex package)
* Getting elixir_uuid (Hex package)
* Getting erlex (Hex package)
* Getting protox (Hex package)
* Getting table_rex (Hex package)
* Getting xla (Hex package)
* Getting elixir_make (Hex package)
After that I executed mix compile which resulted in this log:
==> elixir_uuid
Compiling 1 file (.ex)
warning: :crypto.hash/2 defined in application :crypto is used by the current application but the current application does not depend on :crypto. To fix this, you must do one of:
1. If :crypto is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs
2. If :crypto is a dependency, make sure it is listed under "def deps" in your mix.exs
3. In case you don't want to add a requirement to :crypto, you may optionally skip this warning by adding [xref: [exclude: [:crypto]]] to your "def project" in mix.exs
Found at 2 locations:
lib/uuid.ex:589: UUID.namebased_uuid/2
lib/uuid.ex:593: UUID.namebased_uuid/2
warning: :crypto.strong_rand_bytes/1 defined in application :crypto is used by the current application but the current application does not depend on :crypto. To fix this, you must do one of:
1. If :crypto is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs
2. If :crypto is a dependency, make sure it is listed under "def deps" in your mix.exs
3. In case you don't want to add a requirement to :crypto, you may optionally skip this warning by adding [xref: [exclude: [:crypto]]] to your "def project" in mix.exs
Found at 3 locations:
lib/uuid.ex:383: UUID.uuid4/1
lib/uuid.ex:560: UUID.uuid1_clockseq/0
lib/uuid.ex:583: UUID.uuid1_node/1
Generated elixir_uuid app
==> erlex
Compiling 1 file (.yrl)
src/parser.yrl: Warning: conflicts: 27 shift/reduce, 0 reduce/reduce
Compiling 1 file (.xrl)
Compiling 2 files (.erl)
Compiling 1 file (.ex)
Generated erlex app
==> nx
Compiling 20 files (.ex)
Generated nx app
==> dialyxir
Compiling 58 files (.ex)
Generated dialyxir app
==> elixir_make
Compiling 1 file (.ex)
Generated elixir_make app
==> xla
Compiling 2 files (.ex)
Generated xla app
==> exla
Unpacking /home/zeio/.cache/xla/0.2.0/cache/download/xla_extension-x86_64-linux-cpu.tar.gz into /home/zeio/grapex/deps/exla/exla/cache
mkdir -p /home/zeio/grapex/_build/dev/lib/exla/priv
ln -sf /home/zeio/grapex/deps/exla/exla/cache/xla_extension/lib /home/zeio/grapex/_build/dev/lib/exla/priv/lib
g++ -fPIC -I/usr/lib/erlang/erts-12.1.3/include -isystem cache/xla_extension/include -O3 -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -shared -std=c++14 c_src/exla/exla.cc c_src/exla/exla_nif_util.cc c_src/exla/exla_client.cc -o /home/zeio/grapex/_build/dev/lib/exla/priv/libexla.so -L/home/zeio/grapex/_build/dev/lib/exla/priv/lib -lxla_extension -Wl,-rpath,'$ORIGIN/lib'
c_src/exla/exla_client.cc: In function ‘tensorflow::StatusOr<std::vector<exla::ExlaBuffer*> > exla::UnpackRunArguments(ErlNifEnv*, ERL_NIF_TERM, exla::ExlaClient*, int)’:
c_src/exla/exla_client.cc:95:19: warning: redundant move in return statement [-Wredundant-move]
95 | return std::move(arg_buffers);
| ~~~~~~~~~^~~~~~~~~~~~~
c_src/exla/exla_client.cc:95:19: note: remove ‘std::move’ call
Compiling 21 files (.ex)
warning: Nx.Defn.default_options/0 is undefined or private
lib/exla/device_backend.ex:78: EXLA.DeviceBackend.default_client_name/0
warning: Nx.Defn.Composite.reduce/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
lib/exla/defn/stream.ex:95: Nx.Stream.EXLA.Defn.Stream.nx_to_io/1
warning: Nx.Defn.global_default_options/1 is undefined or private
lib/exla.ex:211: EXLA.set_preferred_defn_options/1
warning: Nx.Defn.Composite.traverse/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
lib/exla/defn/buffers.ex:9: EXLA.Defn.Buffers.to_nx!/3
warning: Nx.byte_size/1 is undefined or private
Found at 2 locations:
lib/exla/defn/buffers.ex:17: EXLA.Defn.Buffers.buffer_to_data/2
lib/exla/defn/buffers.ex:18: EXLA.Defn.Buffers.buffer_to_data/2
warning: Nx.Defn.Composite.flatten_list/1 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
Found at 2 locations:
lib/exla/defn.ex:17: EXLA.Defn.__stream__/6
lib/exla/defn.ex:19: EXLA.Defn.__stream__/6
warning: Nx.Defn.Composite.reduce/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
Found at 2 locations:
lib/exla/defn.ex:331: EXLA.Defn.used_inputs_and_hooks/1
lib/exla/defn.ex:366: EXLA.Defn.recur_flatten/3
warning: Nx.Defn.Composite.traverse/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
lib/exla/defn.ex:1405: EXLA.Defn.to_if_branch/5
warning: Nx.Defn.Tree.apply_args/3 is undefined or private
Found at 4 locations:
lib/exla/defn.ex:345: EXLA.Defn.used_inputs_and_hooks/2
lib/exla/defn.ex:467: EXLA.Defn.cached_recur_operator/4
lib/exla/defn.ex:1372: EXLA.Defn.collect_ids/2
lib/exla/defn.ex:1397: EXLA.Defn.collect_args/3
Generated exla app
==> protox
Compiling 22 files (.ex)
Generated protox app
==> optimus
Compiling 17 files (.ex)
Generated optimus app
==> table_rex
Compiling 7 files (.ex)
Generated table_rex app
==> axon
Compiling 20 files (.ex)
Generated axon app
==> axon_onnx
Compiling 3 files (.ex)
Generated axon_onnx app
==> grapex
Compiling 8 files (.ex)
warning: variable "verbose" is unused (if the variable is not meant to be used, prefix it with an underscore)
lib/grapex/models/se.ex:60: Grapex.Model.Se.compute_score/2
warning: function fix_shape/1 is unused
lib/grapex/models/se.ex:31
warning: function multiply/2 is unused
lib/grapex/models/se.ex:46
warning: TranseHeterogenous.save/1 is undefined (module TranseHeterogenous is not available or is yet to be defined)
lib/grapex.ex:299: Grapex.main/1
warning: TranseHeterogenous.test_or_validate/1 is undefined (module TranseHeterogenous is not available or is yet to be defined)
lib/grapex.ex:298: Grapex.main/1
warning: TranseHeterogenous.train_or_import/1 is undefined (module TranseHeterogenous is not available or is yet to be defined)
lib/grapex.ex:297: Grapex.main/1
Generated grapex app
It seems that exla unpacks cached version of the library compiled for cpu (xla_extension-x86_64-linux-cpu.tar.gz) which was downloaded before I started my attempts to run models on gpu. That’s why I tried to remove the ~/.cache/xla folder and run the compilation again, but the output didn’t change significantly:
==> elixir_uuid
Compiling 1 file (.ex)
warning: :crypto.hash/2 defined in application :crypto is used by the current application but the current application does not depend on :crypto. To fix this, you must do one of:
1. If :crypto is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs
2. If :crypto is a dependency, make sure it is listed under "def deps" in your mix.exs
3. In case you don't want to add a requirement to :crypto, you may optionally skip this warning by adding [xref: [exclude: [:crypto]]] to your "def project" in mix.exs
Found at 2 locations:
lib/uuid.ex:589: UUID.namebased_uuid/2
lib/uuid.ex:593: UUID.namebased_uuid/2
warning: :crypto.strong_rand_bytes/1 defined in application :crypto is used by the current application but the current application does not depend on :crypto. To fix this, you must do one of:
1. If :crypto is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs
2. If :crypto is a dependency, make sure it is listed under "def deps" in your mix.exs
3. In case you don't want to add a requirement to :crypto, you may optionally skip this warning by adding [xref: [exclude: [:crypto]]] to your "def project" in mix.exs
Found at 3 locations:
lib/uuid.ex:383: UUID.uuid4/1
lib/uuid.ex:560: UUID.uuid1_clockseq/0
lib/uuid.ex:583: UUID.uuid1_node/1
Generated elixir_uuid app
==> erlex
Compiling 2 files (.erl)
Compiling 1 file (.ex)
Generated erlex app
==> nx
Compiling 20 files (.ex)
Generated nx app
==> dialyxir
Compiling 58 files (.ex)
Generated dialyxir app
==> elixir_make
Compiling 1 file (.ex)
Generated elixir_make app
==> xla
Compiling 2 files (.ex)
Generated xla app
18:22:49.977 [info] Found a matching archive (xla_extension-x86_64-linux-cpu.tar.gz), going to download it
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 676 100 676 0 0 1867 0 --:--:-- --:--:-- --:--:-- 1862
100 91.9M 100 91.9M 0 0 8727k 0 0:00:10 0:00:10 --:--:-- 10.2M
18:23:00.782 [info] Successfully downloaded the XLA archive
==> exla
mkdir -p /home/zeio/grapex/_build/dev/lib/exla/priv
ln -sf /home/zeio/grapex/deps/exla/exla/cache/xla_extension/lib /home/zeio/grapex/_build/dev/lib/exla/priv/lib
g++ -fPIC -I/usr/lib/erlang/erts-12.1.3/include -isystem cache/xla_extension/include -O3 -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -shared -std=c++14 c_src/exla/exla.cc c_src/exla/exla_nif_util.cc c_src/exla/exla_client.cc -o /home/zeio/grapex/_build/dev/lib/exla/priv/libexla.so -L/home/zeio/grapex/_build/dev/lib/exla/priv/lib -lxla_extension -Wl,-rpath,'$ORIGIN/lib'
c_src/exla/exla_client.cc: In function ‘tensorflow::StatusOr<std::vector<exla::ExlaBuffer*> > exla::UnpackRunArguments(ErlNifEnv*, ERL_NIF_TERM, exla::ExlaClient*, int)’:
c_src/exla/exla_client.cc:95:19: warning: redundant move in return statement [-Wredundant-move]
95 | return std::move(arg_buffers);
| ~~~~~~~~~^~~~~~~~~~~~~
c_src/exla/exla_client.cc:95:19: note: remove ‘std::move’ call
Compiling 21 files (.ex)
warning: Nx.Defn.Composite.reduce/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
lib/exla/defn/stream.ex:95: Nx.Stream.EXLA.Defn.Stream.nx_to_io/1
warning: Nx.Defn.Composite.traverse/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
lib/exla/defn/buffers.ex:9: EXLA.Defn.Buffers.to_nx!/3
warning: Nx.Defn.Composite.flatten_list/1 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
Found at 2 locations:
lib/exla/defn.ex:17: EXLA.Defn.__stream__/6
lib/exla/defn.ex:19: EXLA.Defn.__stream__/6
warning: Nx.Defn.Composite.reduce/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
Found at 2 locations:
lib/exla/defn.ex:331: EXLA.Defn.used_inputs_and_hooks/1
lib/exla/defn.ex:366: EXLA.Defn.recur_flatten/3
warning: Nx.Defn.Composite.traverse/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
lib/exla/defn.ex:1405: EXLA.Defn.to_if_branch/5
warning: Nx.Defn.default_options/0 is undefined or private
lib/exla/device_backend.ex:78: EXLA.DeviceBackend.default_client_name/0
warning: Nx.Defn.global_default_options/1 is undefined or private
lib/exla.ex:211: EXLA.set_preferred_defn_options/1
warning: Nx.Defn.Tree.apply_args/3 is undefined or private
Found at 4 locations:
lib/exla/defn.ex:345: EXLA.Defn.used_inputs_and_hooks/2
lib/exla/defn.ex:467: EXLA.Defn.cached_recur_operator/4
lib/exla/defn.ex:1372: EXLA.Defn.collect_ids/2
lib/exla/defn.ex:1397: EXLA.Defn.collect_args/3
warning: Nx.byte_size/1 is undefined or private
Found at 2 locations:
lib/exla/defn/buffers.ex:17: EXLA.Defn.Buffers.buffer_to_data/2
lib/exla/defn/buffers.ex:18: EXLA.Defn.Buffers.buffer_to_data/2
Generated exla app
==> protox
Compiling 22 files (.ex)
Generated protox app
==> optimus
Compiling 17 files (.ex)
Generated optimus app
==> table_rex
Compiling 7 files (.ex)
Generated table_rex app
==> axon
Compiling 20 files (.ex)
Generated axon app
==> axon_onnx
Compiling 3 files (.ex)
Generated axon_onnx app
==> grapex
Compiling 8 files (.ex)
warning: variable "verbose" is unused (if the variable is not meant to be used, prefix it with an underscore)
lib/grapex/models/se.ex:60: Grapex.Model.Se.compute_score/2
warning: function fix_shape/1 is unused
lib/grapex/models/se.ex:31
warning: function multiply/2 is unused
lib/grapex/models/se.ex:46
warning: TranseHeterogenous.save/1 is undefined (module TranseHeterogenous is not available or is yet to be defined)
lib/grapex.ex:299: Grapex.main/1
warning: TranseHeterogenous.test_or_validate/1 is undefined (module TranseHeterogenous is not available or is yet to be defined)
lib/grapex.ex:298: Grapex.main/1
warning: TranseHeterogenous.train_or_import/1 is undefined (module TranseHeterogenous is not available or is yet to be defined)
lib/grapex.ex:297: Grapex.main/1
Generated grapex app
So it just downloaded the same version of the library even though my env variable XLA_TARGET accepts value cuda111. Then I tried to change this value from cuda111 to just cuda, delete the _build and ~/.cache/xla folders and compile again, which produced a runtime error:
==> elixir_uuid
Compiling 1 file (.ex)
warning: :crypto.hash/2 defined in application :crypto is used by the current application but the current application does not depend on :crypto. To fix this, you must do one of:
1. If :crypto is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs
2. If :crypto is a dependency, make sure it is listed under "def deps" in your mix.exs
3. In case you don't want to add a requirement to :crypto, you may optionally skip this warning by adding [xref: [exclude: [:crypto]]] to your "def project" in mix.exs
Found at 2 locations:
lib/uuid.ex:589: UUID.namebased_uuid/2
lib/uuid.ex:593: UUID.namebased_uuid/2
warning: :crypto.strong_rand_bytes/1 defined in application :crypto is used by the current application but the current application does not depend on :crypto. To fix this, you must do one of:
1. If :crypto is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs
2. If :crypto is a dependency, make sure it is listed under "def deps" in your mix.exs
3. In case you don't want to add a requirement to :crypto, you may optionally skip this warning by adding [xref: [exclude: [:crypto]]] to your "def project" in mix.exs
Found at 3 locations:
lib/uuid.ex:383: UUID.uuid4/1
lib/uuid.ex:560: UUID.uuid1_clockseq/0
lib/uuid.ex:583: UUID.uuid1_node/1
Generated elixir_uuid app
==> erlex
Compiling 2 files (.erl)
Compiling 1 file (.ex)
Generated erlex app
==> nx
Compiling 20 files (.ex)
Generated nx app
==> dialyxir
Compiling 58 files (.ex)
Generated dialyxir app
==> elixir_make
Compiling 1 file (.ex)
Generated elixir_make app
==> xla
Compiling 2 files (.ex)
Generated xla app
==> exla
could not compile dependency :exla, "mix compile" failed. You can recompile this dependency with "mix deps.compile exla", update it with "mix deps.update exla" or clean it with "mix deps.clean exla"
** (RuntimeError) none of the precompiled archives matches your target
Expected:
* xla_extension-x86_64-linux-cuda.tar.gz
Found:
* xla_extension-aarch64-darwin-cpu.tar.gz
* xla_extension-aarch64-linux-cpu.tar.gz
* xla_extension-aarch64-linux-cuda102.tar.gz
* xla_extension-x86_64-darwin-cpu.tar.gz
* xla_extension-x86_64-linux-cpu.tar.gz
* xla_extension-x86_64-linux-cuda102.tar.gz
* xla_extension-x86_64-linux-cuda110.tar.gz
* xla_extension-x86_64-linux-cuda111.tar.gz
* xla_extension-x86_64-linux-tpu.tar.gz
You can compile XLA locally by setting an environment variable: XLA_BUILD=true
(xla 0.2.0) lib/xla.ex:171: XLA.download_matching!/1
(xla 0.2.0) lib/xla.ex:33: XLA.archive_path!/0
/home/zeio/grapex/deps/exla/exla/mix.exs:73: EXLA.MixProject.compile/1
(mix 1.12.2) lib/mix/task.ex:458: Mix.Task.run_alias/5
(mix 1.12.2) lib/mix/tasks/compile.all.ex:92: Mix.Tasks.Compile.All.run_compiler/2
(mix 1.12.2) lib/mix/tasks/compile.all.ex:72: Mix.Tasks.Compile.All.compile/4
(mix 1.12.2) lib/mix/tasks/compile.all.ex:59: Mix.Tasks.Compile.All.with_logger_app/2
(mix 1.12.2) lib/mix/tasks/compile.all.ex:36: Mix.Tasks.Compile.All.run/1
Clearly during the previous compilation for some reason it didn’t want to download archive xla_extension-x86_64-linux-cuda111.tar.gz and still preferred to use the version for cpu.
0
zeio
Hmm, actually something strange happens. I’ve edited the source code for xla library (added printing the required archive filename in the beginning of function download_matching! which is located in file deps/xla/lib/xla.ex). Then I changed XLA_TARGET back to cuda111 and run mix compile exla. This time the correct archive was downloaded according to logs:
/home/zeio/.cache/xla/0.2.0/cache/download/xla_extension-x86_64-linux-cuda111.tar.gz
18:57:12.780 [info] Found a matching archive (xla_extension-x86_64-linux-cuda111.tar.gz), going to download it
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 680 100 680 0 0 2054 0 --:--:-- --:--:-- --:--:-- 2054
100 158M 100 158M 0 0 6515k 0 0:00:24 0:00:24 --:--:-- 6346k
18:57:37.699 [info] Successfully downloaded the XLA archive
==> exla
Unpacking /home/zeio/.cache/xla/0.2.0/cache/download/xla_extension-x86_64-linux-cuda111.tar.gz into /home/zeio/grapex/deps/exla/exla/cache
mkdir -p /home/zeio/grapex/_build/dev/lib/exla/priv
ln -sf /home/zeio/grapex/deps/exla/exla/cache/xla_extension/lib /home/zeio/grapex/_build/dev/lib/exla/priv/lib
g++ -fPIC -I/usr/lib/erlang/erts-12.1.3/include -isystem cache/xla_extension/include -O3 -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-comment -shared -std=c++14 c_src/exla/exla.cc c_src/exla/exla_nif_util.cc c_src/exla/exla_client.cc -o /home/zeio/grapex/_build/dev/lib/exla/priv/libexla.so -L/home/zeio/grapex/_build/dev/lib/exla/priv/lib -lxla_extension -Wl,-rpath,'$ORIGIN/lib'
c_src/exla/exla_client.cc: In function ‘tensorflow::StatusOr<std::vector<exla::ExlaBuffer*> > exla::UnpackRunArguments(ErlNifEnv*, ERL_NIF_TERM, exla::ExlaClient*, int)’:
c_src/exla/exla_client.cc:95:19: warning: redundant move in return statement [-Wredundant-move]
95 | return std::move(arg_buffers);
| ~~~~~~~~~^~~~~~~~~~~~~
c_src/exla/exla_client.cc:95:19: note: remove ‘std::move’ call
Compiling 21 files (.ex)
warning: Nx.Defn.Composite.reduce/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
lib/exla/defn/stream.ex:95: Nx.Stream.EXLA.Defn.Stream.nx_to_io/1
warning: Nx.Defn.global_default_options/1 is undefined or private
lib/exla.ex:211: EXLA.set_preferred_defn_options/1
warning: Nx.Defn.Composite.flatten_list/1 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
Found at 2 locations:
lib/exla/defn.ex:17: EXLA.Defn.__stream__/6
lib/exla/defn.ex:19: EXLA.Defn.__stream__/6
warning: Nx.Defn.Composite.reduce/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
Found at 2 locations:
lib/exla/defn.ex:331: EXLA.Defn.used_inputs_and_hooks/1
lib/exla/defn.ex:366: EXLA.Defn.recur_flatten/3
warning: Nx.Defn.default_options/0 is undefined or private
lib/exla/device_backend.ex:78: EXLA.DeviceBackend.default_client_name/0
warning: Nx.Defn.Composite.traverse/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
lib/exla/defn.ex:1405: EXLA.Defn.to_if_branch/5
warning: Nx.Defn.Tree.apply_args/3 is undefined or private
Found at 4 locations:
lib/exla/defn.ex:345: EXLA.Defn.used_inputs_and_hooks/2
lib/exla/defn.ex:467: EXLA.Defn.cached_recur_operator/4
lib/exla/defn.ex:1372: EXLA.Defn.collect_ids/2
lib/exla/defn.ex:1397: EXLA.Defn.collect_args/3
warning: Nx.Defn.Composite.traverse/3 is undefined (module Nx.Defn.Composite is not available or is yet to be defined)
lib/exla/defn/buffers.ex:9: EXLA.Defn.Buffers.to_nx!/3
warning: Nx.byte_size/1 is undefined or private
Found at 2 locations:
lib/exla/defn/buffers.ex:17: EXLA.Defn.Buffers.buffer_to_data/2
lib/exla/defn/buffers.ex:18: EXLA.Defn.Buffers.buffer_to_data/2
Generated exla app
Then I run my script via mix run main.exs which contains model training, and it seems that the process was really happenining on the gpu which I was able to see in the output of the nvidia-smi command:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03 Driver Version: 460.91.03 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1650 Off | 00000000:01:00.0 Off | N/A |
| N/A 50C P8 3W / N/A | 695MiB / 3911MiB | 32% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1760 G /usr/lib/xorg/Xorg 45MiB |
| 0 N/A N/A 3295 G /usr/lib/xorg/Xorg 375MiB |
| 0 N/A N/A 3478 G /usr/bin/gnome-shell 70MiB |
| 0 N/A N/A 1225965 G ...AAAAAAAAA= --shared-files 64MiB |
| 0 N/A N/A 1600138 G ...AAAAAAAAA= --shared-files 32MiB |
| 0 N/A N/A 1600174 G ...AAAAAAAA== --shared-files 33MiB |
| 0 N/A N/A 1624190 C .../erts-12.1.3/bin/beam.smp 55MiB |
+-----------------------------------------------------------------------------+
At some moment I stopped the execution, the full output of my script at this point:
Compiling 1 file (.ex)
{:ok, %{'CUDA' => 1, 'Host' => 12}}
{:ok, #Reference<0.3594860528.3707371527.100288>}
18:59:57.763 [info] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
18:59:57.766 [info] XLA service 0x7f30e4495730 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
18:59:57.766 [info] StreamExecutor device (0): GeForce GTX 1650, Compute Capability 7.5
18:59:57.766 [info] Using BFC allocator.
18:59:57.767 [info] XLA backend will use up to 3376283648 bytes on device 0 for BFCAllocator.
Input Files Path : /home/zeio/relentness/Assets/Corpora/Demo/0000/
Setting bern flag to false
---------------------------------------------------------------------------------------------
Model
=============================================================================================
Layer Shape Parameters
=============================================================================================
input_1 ( input ) {nil, 4, 2} 0
embedding_3 ( embedding ) {nil, 4, 2, 10} 580
reshape_4 ( reshape ) {nil, 4, 2, 1, 1, 10} 0
pad_5 ( pad ) {nil, 4, 2, 2, 10, 10} 0
input_6 ( input ) {nil, 4, 1} 0
embedding_8 ( embedding ) {nil, 4, 1, 200} 400
reshape_9 ( reshape ) {nil, 4, 1, 2, 10, 10} 0
concatenate_10 ( concatenate ["pad_5", "reshape_9"] ) {nil, 4, 3, 2, 10, 10} 0
---------------------------------------------------------------------------------------------
The model will not be saved during training because n-export-steps parameter has not been provided.
Epoch: 23, Batch: 10, Loss: 3.55707 ^C
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
(l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
The second and third lines contain output of the following two commands respectively:
IO.inspect EXLA.NIF.get_supported_platforms()
IO.inspect EXLA.NIF.get_gpu_client(1.0, 0)
After that I tried to run the same command again (mix run main.exs), but this time the process has started on cpu instead of gpu, I’ve got the following output:
{:ok, %{'Host' => 12}}
{:error,
'Could not find registered platform with name: "cuda". Available platform names are: Host'}
Input Files Path : /home/zeio/relentness/Assets/Corpora/Demo/0000/
Setting bern flag to false
---------------------------------------------------------------------------------------------
Model
=============================================================================================
Layer Shape Parameters
=============================================================================================
input_1 ( input ) {nil, 4, 2} 0
embedding_3 ( embedding ) {nil, 4, 2, 10} 580
reshape_4 ( reshape ) {nil, 4, 2, 1, 1, 10} 0
pad_5 ( pad ) {nil, 4, 2, 2, 10, 10} 0
input_6 ( input ) {nil, 4, 1} 0
embedding_8 ( embedding ) {nil, 4, 1, 200} 400
reshape_9 ( reshape ) {nil, 4, 1, 2, 10, 10} 0
concatenate_10 ( concatenate ["pad_5", "reshape_9"] ) {nil, 4, 3, 2, 10, 10} 0
---------------------------------------------------------------------------------------------
The model will not be saved during training because n-export-steps parameter has not been provided.
Epoch: 15, Batch: 6, Loss: 4.28919 ^C
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
(l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
And any subsequent run behaves in the similar way - it cannot detect cuda client and use gpu anymore, always running computations on the cpu.
0
josevalim
Creator of Elixir