k-Wave

1. Julien S.
   

   Member
   Joined: Dec '16
   Posts: 3

   Hi,

   I am trying to use the kspaceFirstOrdre3D-CUDA code to simulate non-linear acoustics on a Titan X GPU. With my old graphic card GeForce GT 730 the code worked even if the GPU was not powerfull enough to speed up the execution. But now with my new Titan X card, I systematically get that error when I try to launch the code :
   "Error : All CUDA-capable devices are busy or unavailable".

   I read in the K-wave documentation that the CUDA code is complied for Maxwell graphic cards. My Titan X has a Pascal architecture. Is it why the code execution returns that error ? If so, is it possible to make the CUDA code work with a Pascal card, and how to do so ?

   Thanks in advance for your reply.

   Posted 7 years ago #

2. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,643

   Hi Julien,

   Are you using linux? We've just uploaded a GPU binary for Pascal GPUs. Let me know if it doesn't work.

   Brad.

   Posted 7 years ago #

3. Julien S.
   

   Member
   Joined: Dec '16
   Posts: 3

   Hi Brad,

   First of all, thank you for your reply and for the new version of your GPU code. Indeed I'm using Linux and I've tried your new version of the code for Pascal GPUs. I have actually one problem when running some simulations. When I increase the number of gridpoints I get this error :

   Error : Failed to execute an cuFFT on the GPU for Execute_FFT_3D_R2C.

   For example, with a grid of size 800 x 256 x 256, the code runs properly, but when designing a grid of size 900 x 256 x 256, the code does not start to run and returns this error message.

   Do you have any idea of what the problem could be ?

   Julien

   Posted 7 years ago #

4. Jiri Jaros
   

   Developer
   Joined: Feb '12
   Posts: 115

   Hi Julien,
   this is going to be an out-of-memory problem. cuFFT (CUDA 3D FFT) needs some scratch place to calculate FFTs in a fast way and I think there's not enough space for it.
   This domain size in a fully heterogeneous absorbing case consumes about 6.7 GB of RAM for raw data. FFT can take another 1GB, so if you have less than 8GB, you won't be able to run such a big simulation.

   Jiri

   Posted 7 years ago #

5. Julien S.
   

   Member
   Joined: Dec '16
   Posts: 3

   Hi Jiri,

   With my Titan X GPU, I have theoretically 12288MB of total memory and 12189MB of total dedicated memory (when checking nvidia-settings). I'm running nonlinear simulations in a homogeneous medium. I ran some tests to see what is the limit of gridsize I can reach and what are the corresponding computational resources displayed by k-wave when the cuda code initializes.

   For a grid of total size (PML included) 960 x 240 x 240 the code runs and I obtain :
   - Current host memory in use : 4029MB
   - Current device memory in use : 4352MB

   For a grid of size 1160 x 240 x 240 the code runs and I obtain :
   - Current host memory in use : 4800MB
   - Current device memory in use : 5212MB

   For a grid of size 1360 x 240 x 240 the error message appears and I obtain :
   - Current host memory in use : 5569MB
   - Current device memory in use : 6074MB

   For bigger gridsizes the code does not run.

   If FFT takes another 1GB to run the simulation, my 12000MB GPU still should have enough memory to run a bigger simulation, isn't it ? It seems that the code does not run anymore when I exceed 50% of used dedicated memory (i.e. 6000MB).

   Thank you again,

   Julien

   Posted 7 years ago #

6. Jiri Jaros
   

   Developer
   Joined: Feb '12
   Posts: 115

   Hi Julien,
   I run a couple of tests and it appears the error only occurs on Pascal GPUs. It is likely to be caused by the fact the code was compiled by CUDA 7.5 and Pascal natively supports CUDA 8.0.

   I will have to take a closer look at the problem next week.

   Jiri

   Posted 7 years ago #

7. DLam
   

   Member
   Joined: Oct '16
   Posts: 19

   Hi Brad, Jiri,

   Sorry for necro'ing, but this thread seems the most relevant.
   I'm getting the same error of "Error : All CUDA-capable devices are busy or unavailable".

   My cluster's GPU is a Tesla V100 (Volta archi) running Linux (CentOs 7). I'm wondering if it's a build problem on my side as well?

   Posted 5 years ago #

8. qjk
   

   Member
   Joined: Oct '17
   Posts: 7

   DLam, I received exactly the same error message when trying to run using Tesla V100 using AWS. The exact same code and data file ran perfectly on an AWS Tesla K80 instance.

   I was using Ubuntu 18 for this.

   Posted 5 years ago #

9. Jiri Jaros
   

   Developer
   Joined: Feb '12
   Posts: 115

   Hi both,
   I've just answered this question in

   http://www.k-wave.org/forum/topic/running-on-aws-tesla-instances#post-6815

   Simply said, you have to recompile the code with latest CUDA.

   Jiri

   Posted 5 years ago #

RSS feed for this topic

Reply

You must log in to post.