cudaMemcpy error « k-Wave User Forum

k-Wave User Forum » k-Wave Support » Graphics Processing Unit (GPU)

cudaMemcpy error

(10 posts) (5 voices)

Started 5 years ago by jlc4kk
Latest reply from Eric Miller

Member
Joined: Jun '15
Posts: 7

I recently upgrade by GPU from a Titan XP 12 Gb to a Titan RTX 24 Gb and simulations that were able to run on the older GPU seem to be having issues with the new GPU. Here is the message output from MATLAB:

Running k-Wave simulation...
  start time: 10-Nov-2019 15:52:42
  reference sound speed: 2820m/s
  dt: 7.3052ns, t_end: 397.6583us, time steps: 54436
  input grid size: 412 by 236 by 236 grid points (154.5 by 88.5 by 88.5mm)
  maximum supported frequency: 2.1067MHz
  expanding computational grid...
  computational grid size: 432 by 256 by 256 grid points
  precomputation completed in 51.1248s
  saving input files to disk...
  completed in 51.1804s
+---------------------------------------------------------------+
|                 kspaceFirstOrder3D-CUDA v1.2                  |
+---------------------------------------------------------------+
| Reading simulation configuration:                        Done |
| Selected GPU device id:                                     0 |
| GPU device name:                                    TITAN RTX |
| Number of CPU threads:                                      1 |
+---------------------------------------------------------------+
|                      Simulation details                       |
+---------------------------------------------------------------+
| Domain dimensions:                            432 x 256 x 256 |
| Simulation time steps:                                  54436 |
+---------------------------------------------------------------+
|                        Initialization                         |
+---------------------------------------------------------------+
| Memory allocation:                                       Done |
| Data loading:                                            Done |
| Elapsed time:                                           2.49s |
+---------------------------------------------------------------+
| FFT plans creation:                                      Done |
| Pre-processing phase:                                    Done |
| Elapsed time:                                           3.47s |
+---------------------------------------------------------------+
|                    Computational resources                    |
+---------------------------------------------------------------+
| Current host memory in use:                            9858MB |
| Current device memory in use:                         13971MB |
+---------------------------------------------------------------+
|                          Simulation                           |
+----------+----------------+--------------+--------------------+
| Progress |  Elapsed time  |  Time to go  |  Est. finish time  |
+----------+----------------+--------------+--------------------+
+----------+----------------+--------------+--------------------+
+---------------------------------------------------------------+
|            !!! K-Wave experienced a fatal error !!!           |
+---------------------------------------------------------------+
| GPU error: unspecified launch failure routine name:           |
|        cudaMemcpy(mDeviceData, mHostData, mCapacity * sizeof( |
|        float), cudaMemcpyHostToDevice) in file MatrixClasses\ |
|        BaseFloatMatrix.cpp, line 99.                          |
+---------------------------------------------------------------+
|                      Execution terminated                     |
+---------------------------------------------------------------+
Error using h5readc
The HDF5 library encountered an error and produced the following stack trace information:

    H5G_loc_find_cb      object 'Nx' doesn't exist
    H5G_traverse_real    traversal operator failed
    H5G_traverse         internal path traversal failed
    H5G_loc_find         can't find object
    H5Dopen2             not found

Error in h5read (line 58)
[data,var_class] = h5readc(Filename,Dataset,start,count,stride);

Error in kspaceFirstOrder3DC (line 394)
Nx = h5read(output_filename, '/Nx');

Error in kspaceFirstOrder3DG (line 64)
    sensor_data = kspaceFirstOrder3DC(varargin{:}, 'BinaryName', binary_name);

Error in acousticSimulationAllRings (line 127)
sensor_data = kspaceFirstOrder3DG(kgrid, medium, source, sensor, input_args{:});

Error in runAcoustic (line 11)
acousticSimulationAllRings

My nvidia driver version is 26.21.14.4097
GPU: Titan RTX 24 Gb
OS: Windows 10 Home
32 Gb RAM

Do you have any guidance on resolving this issue?

Thank you.

Posted 5 years ago #

jlc4kk

Member
Joined: Jun '15
Posts: 7

Forgot to add that I am using k-Wave 1.2.1

Posted 5 years ago #
Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

Hi jlc4kk,
I suspect the code was not compiled for RTX GPUs. Could you please run this command to see what happens?

./kspaceFirstOrder3D-CUDA --version

Posted 4 years ago #

jlc4kk

Member
Joined: Jun '15
Posts: 7

Here is the output:

+---------------------------------------------------------------+
|                 kspaceFirstOrder3D-CUDA v1.2                  |
+---------------------------------------------------------------+
| Selected GPU device id:                                     0 |
| GPU device name:                                    TITAN RTX |
+---------------------------------------------------------------+
|                       Build information                       |
+---------------------------------------------------------------+
| Build number:     kspaceFirstOrder3D v3.5                     |
| Build date:       Sep  4 2017                                 |
| Build time:       22:06:54                                    |
| Git hash:         992307ffbd4711af502c9be35a1eacfdbc756887    |
+---------------------------------------------------------------+
| Operating system: Windows x64                                 |
| Compiler name:    Visual Studio C++ 1900                      |
+---------------------------------------------------------------+
| CUDA runtime:     8.0                                         |
| CUDA driver:      10.2                                         |
| CUDA code arch:   6.1                                         |
+---------------------------------------------------------------+
| CUDA device id:   0                                           |
| CUDA device name: TITAN RTX                                   |
| CUDA capability:  7.5                                         |
+---------------------------------------------------------------+
| Contact email:    jarosjir@fit.vutbr.cz                       |
| Contact web:      http://www.k-wave.org                       |
+---------------------------------------------------------------+
|       Copyright (C) 2017 Jiri Jaros and Bradley Treeby        |
+---------------------------------------------------------------+

Posted 4 years ago #

Bradley Treeby

Developer
Joined: Mar '10
Posts: 1,643

Can you try with k-Wave V1.3? That should resolve the problem.

Posted 4 years ago #
Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

Yep, the CUDA runtime is too old for TITAN RTX (By the time we compiled that code, TITAN RTX did not exist and naturally couldn't be supported). You need k-Wave V1.3, as Brad suggested, or recompile the code with newer CUDA and include RTX SM architecture in the makefile.

Posted 4 years ago #

ArikHananel

Member
Joined: Aug '17
Posts: 1

I updated to 1.3 and put the pre-compiled binaries in the k-wave binaries folder and checked the version and got:

+---------------------------------------------------------------+
|                  kspaceFirstOrder-CUDA v1.3                   |
+---------------------------------------------------------------+
| Selected GPU device id:                                     0 |
| GPU device name:                                    TITAN RTX |
+---------------------------------------------------------------+
|                       Build information                       |
+---------------------------------------------------------------+
| Build number:     kspaceFirstOrder v3.6                       |
| Build date:       Feb 26 2020                                 |
| Build time:       18:12:32                                    |
| Git hash:         468dc31c2842a7df5f2a07c3a13c16c9b0b2b770    |
+---------------------------------------------------------------+
| Operating system: Windows x64                                 |
| Compiler name:    Visual Studio C++ 1916                      |
| Processor name:   Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz     |
| Instruction set:  Intel AVX                                   |
+---------------------------------------------------------------+
| CUDA runtime:     10.2                                        |
| CUDA driver:      10.2                                        |
| CUDA code arch:   7.5                                         |
+---------------------------------------------------------------+
| CUDA device id:   0                                           |
| CUDA device name: TITAN RTX                                   |
| CUDA capability:  7.5                                         |
+---------------------------------------------------------------+
| Contact email:    jarosjir@fit.vutbr.cz                       |
| Contact web:      http://www.k-wave.org                       |
+---------------------------------------------------------------+
| Copyright (C) 2011-2020 SC@FIT Research Group, BUT, Czech Rep |
+---------------------------------------------------------------+

but I am getting a same error:

Running k-Wave simulation...
start time: 19-Apr-2020 18:04:43
reference sound speed: 2820m/s
dt: 7.3052ns, t_end: 397.6583us, time steps: 54436
input grid size: 412 by 236 by 236 grid points (154.5 by 88.5 by 88.5mm)
maximum supported frequency: 2.1067MHz
expanding computational grid...
computational grid size: 432 by 256 by 256 grid points
precomputation completed in 1min 10.6293s
saving input files to disk...
completed in 2min 13.8809s

+---------------------------------------------------------------+
|                  kspaceFirstOrder-CUDA v1.3                   |
+---------------------------------------------------------------+
| Reading simulation configuration:                        Done |
| Selected GPU device id:                                     0 |
| GPU device name:                                    TITAN RTX |
| Number of CPU threads:                                      1 |
| Processor name:       Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz |
+---------------------------------------------------------------+
|                      Simulation details                       |
+---------------------------------------------------------------+
| Domain dimensions:                            432 x 256 x 256 |
| Medium type:                                               3D |
| Simulation time steps:                                  54436 |
+---------------------------------------------------------------+
|                        Initialization                         |
+---------------------------------------------------------------+
| Memory allocation:                                       Done |
| Data loading:                                            Done |
| Elapsed time:                                           2.90s |
+---------------------------------------------------------------+
| FFT plans creation:                                      Done |
| Pre-processing phase:                                    Done |
| Elapsed time:                                           2.98s |
+---------------------------------------------------------------+
|                    Computational resources                    |
+---------------------------------------------------------------+
| Current host memory in use:                           10177MB |
| Current device memory in use:                         14421MB |
| Expected output file size:                              175MB |
+---------------------------------------------------------------+
|                          Simulation                           |
+----------+----------------+--------------+--------------------+
| Progress |  Elapsed time  |  Time to go  |  Est. finish time  |
+----------+----------------+--------------+--------------------+
+----------+----------------+--------------+--------------------+
+---------------------------------------------------------------+
|            !!! K-Wave experienced a fatal error !!!           |
+---------------------------------------------------------------+
| GPU error: unspecified launch failure routine name:           |
|        cudaMemcpy(mDeviceData, mHostData, mCapacity * sizeof( |
|        float), cudaMemcpyHostToDevice) in file d:\windows     |
|        source\k-wave-fluid-cuda\sources\matrixclasses\        |
|        basefloatmatrix.cpp, line 99.                          |
+---------------------------------------------------------------+
|                      Execution terminated                     |
+---------------------------------------------------------------+

Error using h5readc
The HDF5 library encountered an error and produced the following stack trace information:

H5G_loc_find_cb object 'Nx' doesn't exist
H5G_traverse_real traversal operator failed
H5G_traverse internal path traversal failed
H5G_loc_find can't find object
H5Dopen2 not found

Error in h5read (line 58)
[data,var_class] = h5readc(Filename,Dataset,start,count,stride);

Error in kspaceFirstOrder3DC (line 569)
Nx = h5read(output_filename, '/Nx');

Error in kspaceFirstOrder3DG (line 69)
sensor_data = kspaceFirstOrder3DC(varargin{:});

Posted 4 years ago #

Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

Hi ArikHananel,
It might be a covered out-of-memory error (but might not be the case since it run on older GPUs). Would it be possible to share an input file (or a matlab script to generate it) for this simulation with me? What kind of source do you use?

Posted 4 years ago #
jlc4kk

Member
Joined: Jun '15
Posts: 7

Yes we can share with you a script of our example. I sent it to your fit email. Thank you for your help!

Posted 4 years ago #
Eric Miller

Member
Joined: Nov '20
Posts: 3

Hi Jiri,

I'm now working on this code with ArikHananel and jlc4kk, has there been any more investigation into this?

Posted 4 years ago #

RSS feed for this topic

Reply

You must log in to post.

k-Wave

A MATLAB toolbox for the time-domain simulation of acoustic wave fields

cudaMemcpy error

Reply

A MATLAB toolbox for the time-domain
simulation of acoustic wave fields