k-Wave

1. erlendsriis
   

   Member
   Joined: Jul '16
   Posts: 1

   Hi,

   (I see that someone else in this forum have experienced the same problem with a Pascal GPU.) I am using the CUDA 7.5 binary file. When I run CUDA simulations for small enough grid sizes it works fine, but when I try to run it for a 400x400x400 grid, I get a "Error : Failed to execute an cuFFT on the GPU for Execute_FFT_3D_R2C."

   Regarding memory requirements, the CUDA card should have about 12 GB of memory, and the script prints "Current host memory in use: 4514MB. Current device memory in use: 4957MB".

   I use the following CUDA version: "NVIDIA (R) Cuda compiler driver, Cuda compilation tools, release 7.5, V7.5.17". The CUDA card is a Tesla K40c which has 11439MiB of memory.

   Do you know what might be the problem here?

   Thanks in advance,
   Erlend

   Posted 7 years ago #

2. Jiri Jaros
   

   Developer
   Joined: Feb '12
   Posts: 115

   Hi erlendsriis,
   I also noticed this bug but only on Pascal GPUs. Here, you show it also crashes on K40 which is Kepler I think. I will have to take a closer look at this. However, from the top of my mind, the CUDA FFT parameters are Nx, Ny, Nz and a buffer. So, I'm not sure why it did not work.

   Will give it a try when updating the code for CUDA 8.0

   Jiri

   Posted 7 years ago #

3. sanand
   

   Member
   Joined: Jun '17
   Posts: 1

   Hi Jiri,
   I get this error on two different GPus: GeForce Titan X GPU and Tesla K40c. I was wondering if you had made any progress?
   Thanks!
   Suma

   Posted 7 years ago #

4. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,643

   Hi Suma,

   I'm fairly certain all of these issues have been fixed for k-Wave V1.2, which will be out before the IEEE IUS conference (less than two weeks time).

   Brad.

   Posted 7 years ago #

5. ttiennot
   

   Member
   Joined: Jul '17
   Posts: 5

   Hi,

   I am currently working with the 1.2 version of the GPU code on a GE Force GTE TITAN X.

   With 1.1 version I used to get the error "Failed to execute an cuFFT on the GPU". I do not have this error anymore but I have a new one : "Error: cuFFT failed to allocate GPU or CPU memory while creating plan for 3D complex-to-real fft."

   Do you know what might be the problem?

   Thank you,

   Thomas

   Posted 7 years ago #

6. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,643

   Hi Thomas,

   What are your dimension sizes? If you use the formula to calculate the memory usage given in the user manual (Section 4.9), does this exceed the amount of memory available on the GPU?

   Brad.

   Posted 7 years ago #

7. ttiennot
   

   Member
   Joined: Jul '17
   Posts: 5

   Hi Brad,

   Thank you very much for your quick answer !

   I am working with a 1024*512*256 matrix which explains why this bug occurs I think... Do you have an idea on how i could steel use k-wave with such heavy matrices ?

   Thanks again,

   Thomas

   Posted 7 years ago #

8. Jiri Jaros
   

   Developer
   Joined: Feb '12
   Posts: 115

   Hi Thomas,
   This is the same error as before. I just unified the error messages with Nvidia CUDA. The problem is that the simulation is obviously big. You'd need about ~12-15GB just for simulation data depending on heterogeneity. In your case, you may be able to accommodate the data in GPU memory. However, the CUDA FFT library also needs some scratch place to calculate FFTs in parallel. And that's where the simulation crashed due to out-of-memory problem.

   The only solution for you is the CPU code (or getting P40 GPU with 24GB of memory).

   Jiri

   Posted 7 years ago #

9. ttiennot
   

   Member
   Joined: Jul '17
   Posts: 5

   Hi Jiri,

   Thank you for your answer, I will try to work with the CPU code in the future.

   Thomas

   Posted 7 years ago #

RSS feed for this topic

Reply

You must log in to post.