k-Wave unable to identify GPU « k-Wave User Forum

k-Wave User Forum » k-Wave Support » Graphics Processing Unit (GPU)

k-Wave unable to identify GPU

(6 posts) (3 voices)

Started 4 years ago by smajmuda
Latest reply from Jiri Jaros

smajmuda

Member
Joined: Jan '20
Posts: 2
Hello,

I am getting the following error when kspaceFirstOrder3DG is called in MATLAB on my university's computing cluster, a description of the problem is mentioned below the error message:
```
kspaceFirstOrder3D-CUDA v1.2

Reading simulation configuration:        Done
Selected GPU device id:        Failed

!!! K-Wave experienced a fatal error !!!

 Error: Insufficient CUDA driver version. Install the latest
        drivers.

                      Execution terminated

Error using h5readc
The filename specified was either not found on the MATLAB path or it contains unsupported characters.

Error in h5read (line 58)
[data,var_class] = h5readc(Filename,Dataset,start,count,stride);

Error in kspaceFirstOrder3DC (line 394)
Nx = h5read(output_filename, '/Nx');

Error in kspaceFirstOrder3DG (line 64)
    sensor_data = kspaceFirstOrder3DC(varargin{:}, 'BinaryName', binary_name);

Error in kwave_3d_GPU (line 213)
sensor_data = kspaceFirstOrder3DG(kgrid, medium, transducer, sensor, input_args{:});
```
For context, to get around a previous error involving the CentOS using an older version of glibc, a singularity container by Nvidia is being used to emulate Ubuntu 18.04 on CentOS. The container can see the video card, MATLAB can see the video card and drivers are at the latest version available from Nvidia. The cluster computing support team suggests the possibilities causing the error among: cuda being too new, being run inside a container, other libraries too old/new.

Please note the version of k-Wave being used is 1.2.1.
Following are the details pertaining to the GPU:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000000:18:00.0 Off | 0 |
| N/A 35C P0 56W / 300W | 0MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

>> gpuDevice

ans =

CUDADevice with properties:

Name: 'Tesla V100-SXM2-16GB'
Index: 1
ComputeCapability: '7.0'
SupportsDouble: 1
DriverVersion: 10.2000
ToolkitVersion: 9
MaxThreadsPerBlock: 1024
MaxShmemPerBlock: 49152
MaxThreadBlockSize: [1024 1024 64]
MaxGridSize: [2.1475e+09 65535 65535]
SIMDWidth: 32
TotalMemory: 1.6946e+10
AvailableMemory: 1.6458e+10
MultiprocessorCount: 80
ClockRateKHz: 1530000
ComputeMode: 'Default'
GPUOverlapsTransfers: 1
KernelExecutionTimeout: 0
CanMapHostMemory: 1
DeviceSupported: 1
DeviceSelected: 1

Thank you.
Posted 4 years ago #
Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

Hi smajmuda,
k-Wave 1.2 does not support Volta GPUs. Please download k-Wave 1.3.

Best
Jiri

Posted 4 years ago #
smajmuda

Member
Joined: Jan '20
Posts: 2
Hi Jiri,

Thanks for the prompt response. It turns out that the cluster, does, indeed have v1.3 installed and yet the errors persist. To expand:

Tried to compile using a singularity container of Ubuntu 18 but even explicit calls in the make file it could not find hdf5.h during compile; compiling in the native CentOS 7 we see errors like the below which are likely related to the older version of glibc CentOS uses.
```
skipping incompatible /lib/libm.so when searching for -lm
skipping incompatible /lib/libgcc_s.so.1 when searching for libgcc_s.so.1
skipping incompatible /lib/libpthread.so when searching for -lpthread
skipping incompatible /lib/libc.so when searching for -lc
skipping incompatible /lib/libgcc_s.so.1 when searching for libgcc_s.so.1
```
The cluster support team have also tried running the pre-compiled binaries via a singularity ubuntu container with no luck.

Best regards,
Shubham.
Posted 4 years ago #
Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

Hi smajmuda,
could you try to run the CUDA code with parameter --version. I'd like to see what the code sees :)

it could not find hdf5.h - This is odd, have you installed HDF5 1.10.x and updated the makefile with the path to HDF5-

I run the CUDA code on Centos 7.7 without any problem.

Posted 4 years ago #
lennartverhagen

Member
Joined: May '20
Posts: 4

Hi Jiri,

Am I correct in understand that when the general recommendation in such a situation is to try and recompile the kspace-CUDA binary?

I'm piloting a new HPC cluster and it seems kspace-CUDA v1.3 does not recognise my gpu. Below is some info:

please note that on this system the GNU libc version is a bit old (2.17), but that updating this would require a full OS install, so instead I preload the required library, as such:
LD_PRELOAD=/opt/glibc/2.27/lib/libm.so.6 ./kspaceFirstOrder-CUDA --version

NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

+-----------------------------------------------------------------------------+ | NVIDIA-SMI 410.72 Driver Version: 410.72 CUDA Version: 10.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla P100-PCIE... Off | 00000000:03:00.0 Off | 0 | | N/A 29C P0 25W / 250W | 0MiB / 16280MiB | 0% E. Process | +-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

┌───────────────────────────────────────────────────────────────┐ │ kspaceFirstOrder-CUDA v1.3 │ ├───────────────────────────────────────────────────────────────┤ │ Selected GPU device id: Failed │ ├───────────────────────────────────────────────────────────────┤ │ Build information │ ├───────────────────────────────────────────────────────────────┤ │ Build number: kspaceFirstOrder v3.6 │ │ Build date: Feb 26 2020 │ │ Build time: 21:27:09 │ │ Git hash: 468dc31c2842a7df5f2a07c3a13c16c9b0b2b770 │ ├───────────────────────────────────────────────────────────────┤ │ Operating system: Linux x64 │ │ Compiler name: GNU C++ 7.3.0 │ │ Processor name: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz │ │ Instruction set: Intel AVX │ ├───────────────────────────────────────────────────────────────┤ │ CUDA runtime: N/A │ │ CUDA driver: 10.0 │ │ CUDA code arch: N/A │ ├───────────────────────────────────────────────────────────────┤ │ CUDA device id: N/A │ │ CUDA device name: N/A │ │ CUDA capability: N/A │ ├───────────────────────────────────────────────────────────────┤ │ Contact email: jarosjir@fit.vutbr.cz │ │ Contact web: http://www.k-wave.org │ ├───────────────────────────────────────────────────────────────┤ │ Copyright (C) 2011-2020 SC@FIT Research Group, BUT, Czech Rep │ └───────────────────────────────────────────────────────────────┘

Posted 4 years ago #
Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

Hi lennartverhagen,
As far as I can remember, I compiled the code with CUDA 10.2. If you simply recompile the code with CUDA 10.0, it should work then.

Jiri

Posted 4 years ago #

RSS feed for this topic

Reply

You must log in to post.

k-Wave

A MATLAB toolbox for the time-domain simulation of acoustic wave fields

k-Wave unable to identify GPU

Reply

A MATLAB toolbox for the time-domain
simulation of acoustic wave fields