Running on AWS Tesla instances « k-Wave User Forum

k-Wave User Forum » Using the C++ Code » GPU Binaries

Running on AWS Tesla instances

(20 posts) (7 voices)

Started 5 years ago by qjk
Latest reply from bastg

qjk

Member
Joined: Oct '17
Posts: 7

Hi,

I have been attempting to use k-wave on AWS. I was able to get it running on an instance with a K80 Tesla card. Then I tried a V100 instance, using the V390 Nvidia driver on Ubuntu 18.04. I received the error:

"Error: All CUDA-capable devices are busy or unavailable"

The AWS instance which gave this error is a p3.2xlarge which has an 8-core Xeon CPU and a single V100 Tesla card.

What am I doing wrong here?

Thanks,
Chris

Posted 5 years ago #
Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

Hi Chris,
the binary, you probably have, was compiled with CUDA 8.0 and there was no support for V100 by that time.

Please download the source codes and recompile the binary with CUDA 10.1. In order to add support for Pascal, Volta and Turing GPUs, you also have to modify the makefile.
Just simply add these lines below the similar ones:

--generate-code arch=compute_60,code=sm_60 \
--generate-code arch=compute_61,code=sm_61 \
--generate-code arch=compute_62,code=sm_62 \
--generate-code arch=compute_70,code=sm_70 \
--generate-code arch=compute_72,code=sm_72 \
--generate-code arch=compute_75,code=sm_75 \

Posted 5 years ago #
DLam

Member
Joined: Oct '16
Posts: 19

Thanks, worked for me!

Posted 5 years ago #
qjk

Member
Joined: Oct '17
Posts: 7

Thank you for explaining the solution!

Posted 5 years ago #

Member
Joined: Oct '17
Posts: 7

I finally got round to trying to build from the latest sources on ubuntu 18.04 using LINKING=SEMI. I was able to compile everything but the linker found an undefined reference to 'aec_buffer_encode' (see below). I did "grep -r" in the source to locate this function but did not find it.

Is it saying that the object file sz_compat.o inside the archive libsz.a calls a function 'aec_buffer_encode' which is not found? The same issue happens if I say LINKING=STATIC.

This is still on the K80 system. I wanted to learn how to compile before going to the V100.

Chris

nvcc -Xcompiler="-fopenmp" -Xlinker="-rpath,/usr/lib/x86_64-linux-gnu/hdf5/serial/lib:/lib64:." -std=c++11 -L/usr/lib/x86_64-linux-gnu/hdf5/serial/lib  -L/lib64  --generate-code arch=compute_60,code=sm_60 --generate-code arch=compute_61,code=sm_61 --generate-code arch=compute_62,code=sm_62 --generate-code arch=compute_70,code=sm_70 --generate-code arch=compute_72,code=sm_72  main.o                 \
        Containers/MatrixContainer.o             \
        Containers/MatrixRecord.o                \
        Containers/OutputStreamContainer.o       \
        Hdf5/Hdf5File.o                          \
        Hdf5/Hdf5FileHeader.o                    \
        KSpaceSolver/KSpaceFirstOrder3DSolver.o  \
        KSpaceSolver/SolverCudaKernels.o         \
        Logger/Logger.o                          \
        MatrixClasses/BaseFloatMatrix.o          \
        MatrixClasses/BaseIndexMatrix.o          \
        MatrixClasses/CufftComplexMatrix.o       \
        MatrixClasses/ComplexMatrix.o            \
        MatrixClasses/IndexMatrix.o              \
        MatrixClasses/RealMatrix.o               \
        OutputStreams/BaseOutputStream.o         \
        OutputStreams/IndexOutputStream.o        \
        OutputStreams/CuboidOutputStream.o       \
        OutputStreams/WholeDomainOutputStream.o  \
        OutputStreams/OutputStreamsCudaKernels.o \
        Parameters/CommandLineParameters.o       \
        Parameters/Parameters.o                  \
        Parameters/CudaParameters.o              \
        Parameters/CudaDeviceConstants.o         \
        /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5_hl.a /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.a /usr/lib/x86_64-linux-gnu/libz.a /usr/lib/x86_64-linux-gnu/libsz.a -lcufft -ldl                                       \
        -o kspaceFirstOrder3D-CUDA
/usr/lib/x86_64-linux-gnu/libsz.a(sz_compat.o): In function

SZ_BufftoBuffCompress':
(.text+0x149): undefined reference to `aec_buffer_encode'
(.text+0x339): undefined reference to `aec_buffer_encode'
/usr/lib/x86_64-linux-gnu/libsz.a(sz_compat.o): In function `SZ_BufftoBuffDecompress':
(.text+0x51a): undefined reference to `aec_buffer_decode'
(.text+0x618): undefined reference to `aec_buffer_decode'
collect2: error: ld returned 1 exit status
Makefile:174: recipe for target 'kspaceFirstOrder3D-CUDA' failed
make: *** [kspaceFirstOrder3D-CUDA] Error 1`

Posted 5 years ago #

andrewFrizado

Member
Joined: Nov '18
Posts: 4

Thanks Jiri, I have got kWave running on Turing RTX 2080 Ti thanks to this. I had to also download the szip and zlib library dependancies and add them to the path in the makefile, plus removed

--generate-code arch=compute_20 ...
--generate-code arch=compute_21...

as there was no support in CUDA 10.1 for these architectures (or, at least, that was the error that was the error reported upon compilation)

qjk - your error may be associated with not having the full szip and zlib library dependancies - just a guess though

Posted 5 years ago #
qjk

Member
Joined: Oct '17
Posts: 7

Thanks Andrew,

Understood that the error I am getting must be connected to the compression libraries. I tried to apt-get every package that might be relevant to sz2 and zlib, but without success.

Chris

Posted 5 years ago #
qjk

Member
Joined: Oct '17
Posts: 7

I was able to get the:

(.text+0x618): undefined reference to `aec_buffer_decode'

error to go away by adding $(SZIP_DIR)/libaec.a to the definition of the LIBS variable. Now the linker completes.

I added the --generate-code arch=... lines to the Makefile. Now when I try to run I am back to "Error: All CUDA-capable devices are busy or unavailable."

This could be due to building on a K80 instance with nvcc v9.1. I need to sort out which version of CUDA runs on which hardware. apt-get doesn't seem to want to give me a newer CUDA installation and I'd prefer to use that rather than getting an installation directly from Nvidia since it deals with dependencies.

Posted 5 years ago #
qjk

Member
Joined: Oct '17
Posts: 7

Solved the above problem by making an AWS instance with Ubuntu 19.04. The other one I had was 18.04 which is the version Amazon makes the most obvious. 19.04 "Disco" installs CUDA 10.1 with apt-get.

All is well in V100 land now. Considerably faster than K80.

Posted 5 years ago #
Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

Hi qjk,
When building the codes, I always download sources of SZIP, HDF5 and CUDA and compile them with the same compiler as the k-Wave binary. This saves a lot of problems with different versions, compile parameters, etc.

I run on Ubuntu 18.04 and CUDA 10.1 without any problems (installed manually form Nvidia website).

Best
Jiri

Posted 5 years ago #
andrewFrizado

Member
Joined: Nov '18
Posts: 4
Hi Jiri,

I have also been trying to get the same setup on a PC running Windows 7 with a NVIDIA RTX 2080 Ti. I am able to drop in the binaries for Pascal architecture and get good performance, though not quite as good as the setup on the Linux machine.

I am assuming that this is related to the mismatch of the Turing architecture etc. To workaround, I am trying to recompile the source codes in Visual Studio 2015 (I have also tried in Visual Studio 2013 with v120 platform toolset) but continuously am running into new errors with the library dependencies (directly related to the kWave folders that are in the directory). I have looked at the vcxproj file in a text editor and have updated some of the variables to fit my compilation paramters (HDF1.8.18 instead of HDF5 1.8.19, and CUDA 10.1). My unfamiliarity in Windows and VS definitely lend itself to my troubles, but I was wondering what aleterations I would need to make to the vcxproj file such that I can simply build the code from the provided .sln file. I also assumed that for strictly compiling GPU code the intel compiler and library would not be needed. Is this correct? What else is needed to get these binaries compiled?

Attached are my errors using VS 2015 BuildTools (they are repeated from each k-Wave file in the zip directories):
```
"C:\Users\Andrew\Documents\kWaveSourceTest\k-wave-fluid-cuda.sln" (default targ
et) (1) ->
"C:\Users\Andrew\Documents\kWaveSourceTest\k-wave-fluid-cuda\k-wave-fluid-cuda.
vcxproj" (default target) (2) ->
"C:\Users\Andrew\Documents\kWaveSourceTest\k-wave-fluid-cuda\k-wave-fluid-cuda.
vcxproj" (CudaBuildCore target) (2:2) ->
(CudaBuildCore target) ->
  C:/Users/Andrew/Documents/kWaveSourceTest/k-wave-fluid-cuda/KSpaceSolver/Solv
erCUDAKernels.cu(34): fatal error C1083: Cannot open include file: 'KSpaceSolve
r/SolverCudaKernels.cuh': No such file or directory [C:\Users\Andrew\Documents\
kWaveSourceTest\k-wave-fluid-cuda\k-wave-fluid-cuda.vcxproj]
```
Posted 5 years ago #

Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

Hi andrewFrizado,
Windows is always a bit tricky and it takes me a day to figure out how to compile every new version. I assume you've installed HDF5 library (any version newer than 1.8.10 is fine) and you know where the .dll and .h files are.

Here is a dump from my *.vcxproj. I would recommend to start a new CUDA project, add all source codes in the project and then set up the libraries, folders, etc.

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup Label="ProjectConfigurations">
    <ProjectConfiguration Include="Debug|x64">
      <Configuration>Debug</Configuration>
      <Platform>x64</Platform>
    </ProjectConfiguration>
    <ProjectConfiguration Include="Release|x64">
      <Configuration>Release</Configuration>
      <Platform>x64</Platform>
    </ProjectConfiguration>
  </ItemGroup>
  <ItemGroup>
    <ClCompile Include="..\Sources\Containers\MatrixContainer.cpp" />
    <ClCompile Include="..\Sources\Containers\MatrixRecord.cpp" />
    <ClCompile Include="..\Sources\Containers\OutputStreamContainer.cpp" />
    <ClCompile Include="..\Sources\GetoptWin64\Getopt.cpp" />
    <ClCompile Include="..\Sources\Hdf5\Hdf5File.cpp" />
    <ClCompile Include="..\Sources\Hdf5\Hdf5FileHeader.cpp" />
    <ClCompile Include="..\Sources\KSpaceSolver\KSpaceFirstOrderSolver.cpp" />
    <ClCompile Include="..\Sources\Logger\Logger.cpp" />
    <ClCompile Include="..\Sources\main.cpp" />
    <ClCompile Include="..\Sources\MatrixClasses\BaseFloatMatrix.cpp" />
    <ClCompile Include="..\Sources\MatrixClasses\BaseIndexMatrix.cpp" />
    <ClCompile Include="..\Sources\MatrixClasses\ComplexMatrix.cpp" />
    <ClCompile Include="..\Sources\MatrixClasses\CufftComplexMatrix.cpp" />
    <ClCompile Include="..\Sources\MatrixClasses\IndexMatrix.cpp" />
    <ClCompile Include="..\Sources\MatrixClasses\RealMatrix.cpp" />
    <ClCompile Include="..\Sources\OutputStreams\BaseOutputStream.cpp" />
    <ClCompile Include="..\Sources\OutputStreams\CuboidOutputStream.cpp" />
    <ClCompile Include="..\Sources\OutputStreams\IndexOutputStream.cpp" />
    <ClCompile Include="..\Sources\OutputStreams\WholeDomainOutputStream.cpp" />
    <ClCompile Include="..\Sources\Parameters\CommandLineParameters.cpp" />
    <ClCompile Include="..\Sources\Parameters\CudaParameters.cpp" />
    <ClCompile Include="..\Sources\Parameters\Parameters.cpp" />
  </ItemGroup>
  <ItemGroup>
    <ClInclude Include="..\Sources\Containers\MatrixContainer.h" />
    <ClInclude Include="..\Sources\Containers\MatrixRecord.h" />
    <ClInclude Include="..\Sources\Containers\OutputStreamContainer.h" />
    <ClInclude Include="..\Sources\GetoptWin64\Getopt.h" />
    <ClInclude Include="..\Sources\Hdf5\Hdf5File.h" />
    <ClInclude Include="..\Sources\Hdf5\Hdf5FileHeader.h" />
    <ClInclude Include="..\Sources\KSpaceSolver\KSpaceFirstOrderSolver.h" />
    <ClInclude Include="..\Sources\KSpaceSolver\SolverCudaKernels.cuh" />
    <ClInclude Include="..\Sources\Logger\ErrorMessages.h" />
    <ClInclude Include="..\Sources\Logger\ErrorMessagesLinux.h" />
    <ClInclude Include="..\Sources\Logger\ErrorMessagesWindows.h" />
    <ClInclude Include="..\Sources\Logger\Logger.h" />
    <ClInclude Include="..\Sources\Logger\OutputMessages.h" />
    <ClInclude Include="..\Sources\Logger\OutputMessagesLinux.h" />
    <ClInclude Include="..\Sources\Logger\OutputMessagesWindows.h" />
    <ClInclude Include="..\Sources\MatrixClasses\BaseFloatMatrix.h" />
    <ClInclude Include="..\Sources\MatrixClasses\BaseIndexMatrix.h" />
    <ClInclude Include="..\Sources\MatrixClasses\BaseMatrix.h" />
    <ClInclude Include="..\Sources\MatrixClasses\ComplexMatrix.h" />
    <ClInclude Include="..\Sources\MatrixClasses\CufftComplexMatrix.h" />
    <ClInclude Include="..\Sources\MatrixClasses\IndexMatrix.h" />
    <ClInclude Include="..\Sources\MatrixClasses\RealMatrix.h" />
    <ClInclude Include="..\Sources\OutputStreams\BaseOutputStream.h" />
    <ClInclude Include="..\Sources\OutputStreams\CuboidOutputStream.h" />
    <ClInclude Include="..\Sources\OutputStreams\IndexOutputStream.h" />
    <ClInclude Include="..\Sources\OutputStreams\OutputStreamsCudaKernels.cuh" />
    <ClInclude Include="..\Sources\OutputStreams\WholeDomainOutputStream.h" />
    <ClInclude Include="..\Sources\Parameters\CommandLineParameters.h" />
    <ClInclude Include="..\Sources\Parameters\CudaDeviceConstants.cuh" />
    <ClInclude Include="..\Sources\Parameters\CudaParameters.h" />
    <ClInclude Include="..\Sources\Parameters\Parameters.h" />
    <ClInclude Include="..\Sources\Utils\CudaUtils.cuh" />
    <ClInclude Include="..\Sources\Utils\DimensionSizes.h" />
    <ClInclude Include="..\Sources\Utils\MatrixNames.h" />
    <ClInclude Include="..\Sources\Utils\TimeMeasure.h" />
  </ItemGroup>
  <ItemGroup>
    <CudaCompile Include="..\Sources\KSpaceSolver\SolverCudaKernels.cu" />
    <CudaCompile Include="..\Sources\OutputStreams\OutputStreamsCudaKernels.cu" />
    <CudaCompile Include="..\Sources\Parameters\CudaDeviceConstants.cu" />
  </ItemGroup>
  <PropertyGroup Label="Globals">
    <ProjectGuid>{B040477E-5790-4F5D-B010-B7B57EE26FED}</ProjectGuid>
    <RootNamespace>k_wave_fluid_cuda</RootNamespace>
    <WindowsTargetPlatformVersion>10.0.17763.0</WindowsTargetPlatformVersion>
  </PropertyGroup>
  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
    <ConfigurationType>Application</ConfigurationType>
    <UseDebugLibraries>true</UseDebugLibraries>
    <CharacterSet>MultiByte</CharacterSet>
    <PlatformToolset>v141</PlatformToolset>
    <WholeProgramOptimization>true</WholeProgramOptimization>
  </PropertyGroup>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
    <ConfigurationType>Application</ConfigurationType>
    <UseDebugLibraries>false</UseDebugLibraries>
    <WholeProgramOptimization>true</WholeProgramOptimization>
    <CharacterSet>MultiByte</CharacterSet>
    <PlatformToolset>v141</PlatformToolset>
  </PropertyGroup>
  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
  <ImportGroup Label="ExtensionSettings">
    <Import Project="$(VCTargetsPath)\BuildCustomizations\CUDA 10.0.props" />
  </ImportGroup>
  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
  </ImportGroup>
  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
  </ImportGroup>
  <PropertyGroup Label="UserMacros" />
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    <LinkIncremental>
    </LinkIncremental>
    <IncludePath>$(VC_IncludePath);$(WindowsSDK_IncludePath);c:\Program Files\HDF_Group\HDF5\1.8.21\include;.;..\Sources</IncludePath>
    <LibraryPath>$(VC_LibraryPath_x64);$(WindowsSDK_LibraryPath_x64);$(NETFXKitsDir)Lib\um\x64;c:\Program Files\HDF_Group\HDF5\1.8.21\lib</LibraryPath>
  </PropertyGroup>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
    <IncludePath>$(VC_IncludePath);$(WindowsSDK_IncludePath);c:\Program Files\HDF_Group\HDF5\1.8.21\include;.;..\Sources</IncludePath>
    <LibraryPath>$(VC_LibraryPath_x64);$(WindowsSDK_LibraryPath_x64);$(NETFXKitsDir)Lib\um\x64;c:\Program Files\HDF_Group\HDF5\1.8.21\lib</LibraryPath>
  </PropertyGroup>
  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    <ClCompile>
      <WarningLevel>Level3</WarningLevel>
      <PreprocessorDefinitions>WIN32;WIN64;_DEBUG;_CONSOLE;%(PreprocessorDefinitions);__KWAVE_GIT_HASH__="b6413962eefd51c642c627d5ceff5093c7ad8da4"</PreprocessorDefinitions>
      <FavorSizeOrSpeed>
      </FavorSizeOrSpeed>
      <FloatingPointModel>Precise</FloatingPointModel>
      <LanguageStandard>
      </LanguageStandard>
      <MultiProcessorCompilation>true</MultiProcessorCompilation>
      <EnableEnhancedInstructionSet>
      </EnableEnhancedInstructionSet>
      <OpenMPSupport>false</OpenMPSupport>
      <CompileAs>Default</CompileAs>
      <BasicRuntimeChecks>Default</BasicRuntimeChecks>
      <WholeProgramOptimization>false</WholeProgramOptimization>
      <OpenMP>GenerateParallelCode</OpenMP>
      <CCppSupport>Cpp11Support</CCppSupport>
    </ClCompile>
    <Link>
      <GenerateDebugInformation>true</GenerateDebugInformation>
      <SubSystem>Console</SubSystem>
      <AdditionalDependencies>cudart_static.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies);libszip.lib;libzlib.lib;libhdf5.lib;libhdf5_hl.lib;cufft.lib</AdditionalDependencies>
      <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories);$(CudaToolkitLibDir);</AdditionalLibraryDirectories>
    </Link>
    <CudaCompile>
      <TargetMachinePlatform>64</TargetMachinePlatform>
      <GenerateRelocatableDeviceCode>true</GenerateRelocatableDeviceCode>
      <PtxAsOptionV>true</PtxAsOptionV>
      <FastMath>true</FastMath>
      <CodeGeneration>compute_50,sm_50</CodeGeneration>
      <AdditionalCompilerOptions>
      </AdditionalCompilerOptions>
    </CudaCompile>
  </ItemDefinitionGroup>
  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
    <ClCompile>
      <WarningLevel>Level3</WarningLevel>
      <Optimization>MaxSpeed</Optimization>
      <FunctionLevelLinking>true</FunctionLevelLinking>
      <IntrinsicFunctions>true</IntrinsicFunctions>
      <PreprocessorDefinitions>WIN32;WIN64;NDEBUG;_CONSOLE;%(PreprocessorDefinitions);__KWAVE_GIT_HASH__="b6413962eefd51c642c627d5ceff5093c7ad8da4"</PreprocessorDefinitions>
      <MultiProcessorCompilation>true</MultiProcessorCompilation>
      <EnableEnhancedInstructionSet>
      </EnableEnhancedInstructionSet>
      <LanguageStandard>stdcpp14</LanguageStandard>
      <FloatingPointModel>
      </FloatingPointModel>
      <OpenMPSupport>false</OpenMPSupport>
    </ClCompile>
    <Link>
      <GenerateDebugInformation>false</GenerateDebugInformation>
      <EnableCOMDATFolding>true</EnableCOMDATFolding>
      <OptimizeReferences>true</OptimizeReferences>
      <SubSystem>Console</SubSystem>
      <AdditionalDependencies>cudart_static.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies);libszip.lib;libzlib.lib;libhdf5.lib;libhdf5_hl.lib;cufft.lib</AdditionalDependencies>
    </Link>
    <CudaCompile>
      <TargetMachinePlatform>64</TargetMachinePlatform>
      <GenerateRelocatableDeviceCode>true</GenerateRelocatableDeviceCode>
      <CodeGeneration>compute_75,sm_75</CodeGeneration>
      <FastMath>true</FastMath>
    </CudaCompile>
  </ItemDefinitionGroup>
  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
  <ImportGroup Label="ExtensionTargets">
    <Import Project="$(VCTargetsPath)\BuildCustomizations\CUDA 10.0.targets" />
  </ImportGroup>
</Project>

Posted 5 years ago #

andrewFrizado

Member
Joined: Nov '18
Posts: 4

Hey Jiri,

I followed your instructions as well as getting the original vcxproj to adhere to the correct paths and library paths etc (including the HDF5 libraries) - only to uncover over 100 new errors in the code. Almost all are related to expecting a semicolon (;) or unaware of the "constexp" term in the code. Any tips to get around this?

Here are a few of the 139 errors (same ones repeated multiple times throughout the build):

1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2061): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2070): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2071): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2061): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2061): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2070): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2071): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2061): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2061): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2070): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2071): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2061): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2061): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2070): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2071): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2061): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2061): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2070): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2071): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2061): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2061): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2062): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2070): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2071): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2280): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2280): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2281): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2281): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2300): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2310): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2327): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2336): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2356): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2367): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2384): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2395): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2280): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2280): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2281): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2281): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2300): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2310): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2327): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2336): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2356): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2367): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2384): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2395): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2280): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2280): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2281): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2281): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2300): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2310): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2327): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2336): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2356): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2367): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2384): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2395): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2280): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2280): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2281): warning : expression has no effect
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2281): error : expected a ";"
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2300): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2310): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2327): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2336): error : identifier "outPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2356): error : identifier "inPad" is undefined
1>C:/Users/Andrew/Documents/Test/k-wave-fluid-cuda/KSpaceSolver/SolverCUDAKernels.cu(2367): error : identifier "outPad" is undefined

Thanks for the help!

Andrew

Posted 5 years ago #

Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

Wow!
do you compile it with the NVidia Nsight? It looks like the compiler does not know some CUDA stuff.

Best
Jiri

Posted 5 years ago #
Joehansenshearer

Member
Joined: Nov '19
Posts: 1

Hi there,

I was getting the same errors as qjk. I applied all the suggested fixes and they solved my problems. Unfortunately now I have run into a further errors when I use the MakeFile.

nvcc fatal : A single input file is required for a non-link phase when an outputfile is specified
make: *** [Makefile:214: KSpaceSolver/SolverCudaKernels.o] Error 1
make: *** Waiting for unfinished jobs....
nvcc fatal : A single input file is required for a non-link phase when an outputfile is specified
make: *** [Makefile:220: Parameters/CudaDeviceConstants.o] Error 1
nvcc fatal : A single input file is required for a non-link phase when an outputfile is specified
make: *** [Makefile:217: OutputStreams/OutputStreamsCudaKernels.o] Error 1

Does anyone know how to solve this problem? Any help would be appreciated.

So far I have;
Downloaded source file from http://www.k-wave.org/download.php
Edited MakeFile removing
--generate-code arch=compute_20 ...
--generate-code arch=compute_21...
and added
--generate-code arch=compute_60,code=sm_60 \
--generate-code arch=compute_61,code=sm_61 \
--generate-code arch=compute_62,code=sm_62 \
--generate-code arch=compute_70,code=sm_70 \
--generate-code arch=compute_72,code=sm_72 \
--generate-code arch=compute_75,code=sm_75 \
as suggested.

I have then added appropriate paths for linking.

I had an additional problem where in Hdf5File.h where #include <hdf5.h> was unable to find file so i added full path here
#include "/usr/include/hdf5/serial/hdf5.h"
#include "/usr/include/hdf5/serial/hdf5_hl.h"

Now I am running into the errors specified above when running make.

I am using a GeForce RTX 2060 Super, CUDA 10.1 and Ubuntu 19.10

Thanks for any help in advance.

Regards

Joseph

Posted 5 years ago #
mckao

Member
Joined: Feb '20
Posts: 1

Could K-wave post an updated binary build of the CUDA code?

That would be very much appreciated, thank you

Posted 4 years ago #

Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

I'll do so next week. For now, this is my vcxproj file

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup Label="ProjectConfigurations">
    <ProjectConfiguration Include="Debug|x64">
      <Configuration>Debug</Configuration>
      <Platform>x64</Platform>
    </ProjectConfiguration>
    <ProjectConfiguration Include="Release|x64">
      <Configuration>Release</Configuration>
      <Platform>x64</Platform>
    </ProjectConfiguration>
  </ItemGroup>
  <ItemGroup>
    <ClCompile Include="Containers\MatrixContainer.cpp" />
    <ClCompile Include="Containers\OutputStreamContainer.cpp" />
    <ClCompile Include="GetoptWin64\Getopt.cpp" />
    <ClCompile Include="Hdf5\Hdf5File.cpp" />
    <ClCompile Include="Hdf5\Hdf5FileHeader.cpp" />
    <ClCompile Include="KSpaceSolver\KSpaceFirstOrderSolver.cpp" />
    <ClCompile Include="Logger\Logger.cpp" />
    <ClCompile Include="main.cpp" />
    <ClCompile Include="MatrixClasses\BaseFloatMatrix.cpp" />
    <ClCompile Include="MatrixClasses\BaseIndexMatrix.cpp" />
    <ClCompile Include="MatrixClasses\ComplexMatrix.cpp" />
    <ClCompile Include="MatrixClasses\CufftComplexMatrix.cpp" />
    <ClCompile Include="MatrixClasses\IndexMatrix.cpp" />
    <ClCompile Include="MatrixClasses\RealMatrix.cpp" />
    <ClCompile Include="OutputStreams\BaseOutputStream.cpp" />
    <ClCompile Include="OutputStreams\CuboidOutputStream.cpp" />
    <ClCompile Include="OutputStreams\IndexOutputStream.cpp" />
    <ClCompile Include="OutputStreams\WholeDomainOutputStream.cpp" />
    <ClCompile Include="Parameters\CommandLineParameters.cpp" />
    <ClCompile Include="Parameters\CudaParameters.cpp" />
    <ClCompile Include="Parameters\Parameters.cpp" />
  </ItemGroup>
  <ItemGroup>
    <ClInclude Include="Containers\CudaMatrixContainer.cuh" />
    <ClInclude Include="Containers\MatrixContainer.h" />
    <ClInclude Include="Containers\MatrixRecord.h" />
    <ClInclude Include="Containers\OutputStreamContainer.h" />
    <ClInclude Include="GetoptWin64\Getopt.h" />
    <ClInclude Include="Hdf5\Hdf5File.h" />
    <ClInclude Include="Hdf5\Hdf5FileHeader.h" />
    <ClInclude Include="KSpaceSolver\KSpaceFirstOrderSolver.h" />
    <ClInclude Include="KSpaceSolver\SolverCudaKernels.cuh" />
    <ClInclude Include="Logger\ErrorMessages.h" />
    <ClInclude Include="Logger\ErrorMessagesLinux.h" />
    <ClInclude Include="Logger\ErrorMessagesWindows.h" />
    <ClInclude Include="Logger\Logger.h" />
    <ClInclude Include="Logger\OutputMessages.h" />
    <ClInclude Include="Logger\OutputMessagesLinux.h" />
    <ClInclude Include="Logger\OutputMessagesWindows.h" />
    <ClInclude Include="MatrixClasses\BaseFloatMatrix.h" />
    <ClInclude Include="MatrixClasses\BaseIndexMatrix.h" />
    <ClInclude Include="MatrixClasses\BaseMatrix.h" />
    <ClInclude Include="MatrixClasses\ComplexMatrix.h" />
    <ClInclude Include="MatrixClasses\CufftComplexMatrix.h" />
    <ClInclude Include="MatrixClasses\IndexMatrix.h" />
    <ClInclude Include="MatrixClasses\RealMatrix.h" />
    <ClInclude Include="MatrixClasses\TransposeCudaKernels.cuh" />
    <ClInclude Include="OutputStreams\BaseOutputStream.h" />
    <ClInclude Include="OutputStreams\CuboidOutputStream.h" />
    <ClInclude Include="OutputStreams\IndexOutputStream.h" />
    <ClInclude Include="OutputStreams\OutputStreamsCudaKernels.cuh" />
    <ClInclude Include="OutputStreams\WholeDomainOutputStream.h" />
    <ClInclude Include="Parameters\CommandLineParameters.h" />
    <ClInclude Include="Parameters\CudaDeviceConstants.cuh" />
    <ClInclude Include="Parameters\CudaParameters.h" />
    <ClInclude Include="Parameters\Parameters.h" />
    <ClInclude Include="Utils\CudaUtils.cuh" />
    <ClInclude Include="Utils\DimensionSizes.h" />
    <ClInclude Include="Utils\MatrixNames.h" />
    <ClInclude Include="Utils\TimeMeasure.h" />
  </ItemGroup>
  <ItemGroup>
    <CudaCompile Include="Containers\CudaMatrixContainer.cu" />
    <CudaCompile Include="KSpaceSolver\SolverCudaKernels.cu" />
    <CudaCompile Include="MatrixClasses\TransposeCudaKernels.cu" />
    <CudaCompile Include="OutputStreams\OutputStreamsCudaKernels.cu" />
    <CudaCompile Include="Parameters\CudaDeviceConstants.cu" />
  </ItemGroup>
  <PropertyGroup Label="Globals">
    <ProjectGuid>{B040477E-5790-4F5D-B010-B7B57EE26FED}</ProjectGuid>
    <RootNamespace>k_wave_fluid_cuda</RootNamespace>
    <WindowsTargetPlatformVersion>10.0.17763.0</WindowsTargetPlatformVersion>
  </PropertyGroup>
  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
    <ConfigurationType>Application</ConfigurationType>
    <UseDebugLibraries>true</UseDebugLibraries>
    <CharacterSet>MultiByte</CharacterSet>
    <PlatformToolset>v141</PlatformToolset>
    <WholeProgramOptimization>true</WholeProgramOptimization>
  </PropertyGroup>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
    <ConfigurationType>Application</ConfigurationType>
    <UseDebugLibraries>false</UseDebugLibraries>
    <WholeProgramOptimization>true</WholeProgramOptimization>
    <CharacterSet>MultiByte</CharacterSet>
    <PlatformToolset>v141</PlatformToolset>
    <UseOfMfc>
    </UseOfMfc>
  </PropertyGroup>
  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
  <ImportGroup Label="ExtensionSettings">
    <Import Project="$(VCTargetsPath)\BuildCustomizations\CUDA 10.2.props" />
  </ImportGroup>
  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
  </ImportGroup>
  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
  </ImportGroup>
  <PropertyGroup Label="UserMacros" />
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    <LinkIncremental>
    </LinkIncremental>
    <IncludePath>$(VC_IncludePath);$(WindowsSDK_IncludePath);c:\Program Files\HDF_Group\HDF5\1.10.6\include;.;..\Sources</IncludePath>
    <LibraryPath>$(VC_LibraryPath_x64);$(WindowsSDK_LibraryPath_x64);$(NETFXKitsDir)Lib\um\x64;c:\Program Files\HDF_Group\HDF5\1.10.6\lib</LibraryPath>
  </PropertyGroup>
  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
    <IncludePath>$(VC_IncludePath);$(WindowsSDK_IncludePath);c:\Program Files\HDF_Group\HDF5\1.10.6\include;.;..\Sources</IncludePath>
    <LibraryPath>$(VC_LibraryPath_x64);$(WindowsSDK_LibraryPath_x64);$(NETFXKitsDir)Lib\um\x64;c:\Program Files\HDF_Group\HDF5\1.10.6\lib</LibraryPath>
  </PropertyGroup>
  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    <ClCompile>
      <WarningLevel>Level3</WarningLevel>
      <PreprocessorDefinitions>WIN32;WIN64;_DEBUG;_CONSOLE;%(PreprocessorDefinitions);__KWAVE_GIT_HASH__="b6413962eefd51c642c627d5ceff5093c7ad8da4"</PreprocessorDefinitions>
      <FavorSizeOrSpeed>
      </FavorSizeOrSpeed>
      <FloatingPointModel>Precise</FloatingPointModel>
      <LanguageStandard>
      </LanguageStandard>
      <MultiProcessorCompilation>true</MultiProcessorCompilation>
      <EnableEnhancedInstructionSet>
      </EnableEnhancedInstructionSet>
      <OpenMPSupport>false</OpenMPSupport>
      <CompileAs>Default</CompileAs>
      <BasicRuntimeChecks>Default</BasicRuntimeChecks>
      <WholeProgramOptimization>false</WholeProgramOptimization>
      <OpenMP>GenerateParallelCode</OpenMP>
      <CCppSupport>Cpp11Support</CCppSupport>
    </ClCompile>
    <Link>
      <GenerateDebugInformation>true</GenerateDebugInformation>
      <SubSystem>Console</SubSystem>
      <AdditionalDependencies>cudart_static.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies);libszip.lib;libzlib.lib;libhdf5.lib;libhdf5_hl.lib;cufft.lib</AdditionalDependencies>
      <AdditionalLibraryDirectories>%(AdditionalLibraryDirectories);$(CudaToolkitLibDir);</AdditionalLibraryDirectories>
    </Link>
    <CudaCompile>
      <TargetMachinePlatform>64</TargetMachinePlatform>
      <GenerateRelocatableDeviceCode>true</GenerateRelocatableDeviceCode>
      <PtxAsOptionV>true</PtxAsOptionV>
      <FastMath>true</FastMath>
      <CodeGeneration>
      </CodeGeneration>
      <AdditionalCompilerOptions>
      </AdditionalCompilerOptions>
    </CudaCompile>
  </ItemDefinitionGroup>
  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
    <ClCompile>
      <WarningLevel>Level3</WarningLevel>
      <Optimization>MaxSpeed</Optimization>
      <FunctionLevelLinking>true</FunctionLevelLinking>
      <IntrinsicFunctions>true</IntrinsicFunctions>
      <PreprocessorDefinitions>WIN32;WIN64;NDEBUG;_CONSOLE;%(PreprocessorDefinitions);__KWAVE_GIT_HASH__="5d1e866ae6bd2f83b97f5f98c1081076d2052828"</PreprocessorDefinitions>
      <MultiProcessorCompilation>true</MultiProcessorCompilation>
      <EnableEnhancedInstructionSet>AdvancedVectorExtensions</EnableEnhancedInstructionSet>
      <LanguageStandard>stdcpp14</LanguageStandard>
      <FloatingPointModel>Fast</FloatingPointModel>
      <OpenMPSupport>false</OpenMPSupport>
      <FavorSizeOrSpeed>Speed</FavorSizeOrSpeed>
    </ClCompile>
    <Link>
      <GenerateDebugInformation>false</GenerateDebugInformation>
      <EnableCOMDATFolding>true</EnableCOMDATFolding>
      <OptimizeReferences>true</OptimizeReferences>
      <SubSystem>Console</SubSystem>
      <AdditionalDependencies>cudart_static.lib;kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;%(AdditionalDependencies);libszip.lib;libzlib.lib;libhdf5.lib;libhdf5_hl.lib;cufft.lib</AdditionalDependencies>
      <Version>1.3</Version>
    </Link>
    <CudaCompile>
      <TargetMachinePlatform>64</TargetMachinePlatform>
      <GenerateRelocatableDeviceCode>true</GenerateRelocatableDeviceCode>
      <CodeGeneration>compute_30,sm_30;compute_32,sm_32;compute_35,sm_35;compute_37,sm_37;compute_50,sm_50;compute_52,sm_52;compute_60,sm_60;compute_61,sm_61;compute_70,sm_70;compute_75,sm_75;</CodeGeneration>
      <FastMath>true</FastMath>
      <MaxRegCount />
    </CudaCompile>
  </ItemDefinitionGroup>
  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
  <ImportGroup Label="ExtensionTargets">
    <Import Project="$(VCTargetsPath)\BuildCustomizations\CUDA 10.2.targets" />
  </ImportGroup>
</Project>

Posted 4 years ago #

bastg

Member
Joined: Jun '19
Posts: 7

Hi all -- a follow up on the previous messages of this thread. I have the same error, i.e. "All CUDA-capable devices are busy or unavailable". I have tried to follow the recommendations given in various threads of this forum, but to no avail.

My GPU is a Tesla P100, which is a Pascal architecture with compute capability 60 ("sm_60" or "compute_60"), and therefore requires CUDA 8.0 or newer. I am using the CUDA 10.1, which is almost the latest (10.2 is the latest). This is the CUDA_ARCH variable in my Makefile (entire Makefile is shown at the end of this post, which I have modified a little from the original K-WAVE Makefile):
CUDA_ARCH = --generate-code arch=compute_30,code=sm_30 \
--generate-code arch=compute_32,code=sm_32 \
--generate-code arch=compute_35,code=sm_35 \
--generate-code arch=compute_37,code=sm_37 \
--generate-code arch=compute_50,code=sm_50 \
--generate-code arch=compute_52,code=sm_52 \
--generate-code arch=compute_53,code=sm_53 \
--generate-code arch=compute_60,code=sm_60 \
--generate-code arch=compute_61,code=sm_61 \
--generate-code arch=compute_62,code=sm_62 \
--generate-code arch=compute_70,code=sm_70 \
--generate-code arch=compute_72,code=sm_72 \
--generate-code arch=compute_75,code=sm_75

I am not sure what I am missing... Thanks for your input!

Bastien

************************* ERROR MESSAGE *************************
deepbrain:guerin[152] ../../KWAVE_1.2.1/src/kspaceFirstOrder-CUDA/kspaceFirstOrder-CUDA -i kwave3D_N104_650KHZ_0p75MM_PAR.h5 -o kwave3D_N104_650KHZ_0p75MM_SOL.h5
┌───────────────────────────────────────────────────────────────┐
│ kspaceFirstOrder-CUDA v1.3 │
├───────────────────────────────────────────────────────────────┤
│ Reading simulation configuration: Done │
│ Selected GPU device id: Failed │
└───────────────────────────────────────────────────────────────┘
┌───────────────────────────────────────────────────────────────┐
│ !!! K-Wave experienced a fatal error !!! │
├───────────────────────────────────────────────────────────────┤
│ Error: All CUDA-capable devices are busy or unavailable. │
├───────────────────────────────────────────────────────────────┤
│ Execution terminated │
└───────────────────────────────────────────────────────────────┘
deepbrain:guerin[153]
*****************************************************************

************************* OUTPUT OF NVIDIA-SMI *************************
Fri May 15 10:57:31 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P100-PCIE... Off | 00000000:04:00.0 Off | 0 |
| N/A 33C P0 26W / 250W | 0MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
*****************************************************************

************************* MAKEFILE *************************
# Select compiler
# GNU is default due to Intel 2018's compatibility issues with Ubuntu 18.04
COMPILER = GNU
#COMPILER = Intel

# SEMI static lining is default since it is expected the binary will run on the
# same system.
# Everything will be linked statically, may not work on all GPUs
#LINKING = STATIC
# Everything will be linked dynamically
#LINKING = DYNAMIC
# Everything but CUDA will be linked statically
LINKING = SEMI

# Set up paths: If using modules, the paths are set up automatically,
# otherwise, set paths manually
CUDA_DIR = /usr/pubsw/packages/CUDA/10.1
HDF5_DIR = /autofs/space/guerin/USneuromod/KWAVE_1.2.1/src/hdf5-1.12.0/hdf5
ZLIB_DIR = /autofs/space/guerin/USneuromod/KWAVE_1.2.1/src/zlib-1.2.11
SZIP_DIR = /autofs/space/guerin/USneuromod/KWAVE_1.2.1/src/szip-2.1.1

# Select CPU architecture (what instruction set to be used).
# The native architecture will compile and optimize the code for the underlying
# processor.

CPU_ARCH = native
#CPU_ARCH = AVX
#CPU_ARCH = AVX2
#CPU_ARCH = AVX512

############################### Common flags ###################################
# Git hash of release 1.3
GIT_HASH = -D__KWAVE_GIT_HASH__=\"468dc31c2842a7df5f2a07c3a13c16c9b0b2b770\"

# Replace tabs by spaces
.RECIPEPREFIX +=

# What CUDA GPU architectures to include in the binary
CUDA_ARCH = --generate-code arch=compute_30,code=sm_30 \
--generate-code arch=compute_32,code=sm_32 \
--generate-code arch=compute_35,code=sm_35 \
--generate-code arch=compute_37,code=sm_37 \
--generate-code arch=compute_50,code=sm_50 \
--generate-code arch=compute_52,code=sm_52 \
--generate-code arch=compute_53,code=sm_53 \
--generate-code arch=compute_60,code=sm_60 \
--generate-code arch=compute_61,code=sm_61 \
--generate-code arch=compute_62,code=sm_62 \
--generate-code arch=compute_70,code=sm_70 \
--generate-code arch=compute_72,code=sm_72 \
--generate-code arch=compute_75,code=sm_75

# What libraries to link and how
ifeq ($(LINKING), STATIC)
LDLIBS = $(HDF5_DIR)/lib/libhdf5_hl.a \
$(HDF5_DIR)/lib/libhdf5.a \
$(CUDA_DIR)/lib64/libcufft_static.a \
$(CUDA_DIR)/lib64/libculibos.a \
$(CUDA_DIR)/lib64/libcudart_static.a \
$(ZLIB_DIR)/lib/libz.a \
$(SZIP_DIR)/lib/libsz.a \
-ldl

else ifeq ($(LINKING), DYNAMIC)
LDLIBS = -lhdf5 -lhdf5_hl -lz -lcufft

else ifeq ($(LINKING), SEMI)
LDLIBS = $(HDF5_DIR)/lib/libhdf5_hl.a \
$(HDF5_DIR)/lib/libhdf5.a \
$(ZLIB_DIR)/lib/libz.a \
$(SZIP_DIR)/lib/libsz.a \
-lcufft \
-ldl
endif

############################## NVCC + GNU g++ ##################################
ifeq ($(COMPILER), GNU)
# C++ compiler for CUDA
CXX = /usr/pubsw/packages/CUDA/10.0/bin/nvcc

# C++ standard
CPP_STD = -std=c++11
# Enable OpenMP
OPENMP = -fopenmp

# Set CPU architecture
# Sandy Bridge, Ivy Bridge
ifeq ($(CPU_ARCH), AVX)
CPU_FLAGS = -m64 -mavx

# Haswell, Broadwell
else ifeq ($(CPU_ARCH), AVX2)
CPU_FLAGS = -m64 -mavx2

# Skylake-X, Ice Lake, Cannon Lake
else ifeq ($(CPU_ARCH), AVX512)
CPU_FLAGS = -m64 -mavx512f

# Maximum performance for this CPU
else
CPU_FLAGS = -m64 -march=native -mtune=native
endif

# Use maximum optimization
CPU_OPT = -O3 -ffast-math -fassociative-math
# Use maximum optimization
GPU_OPT = -O3

# CPU Debug flags
CPU_DEBUG =
# Debug flags
GPU_DEBUG =
# Profile flags
PROFILE =
# C++ warning flags
WARNING = -Wall

# Add include directories
INCLUDES = -I$(HDF5_DIR)/include -I.
# Add library directories
LIB_PATHS = -L$(HDF5_DIR)/lib -L$(CUDA_DIR)/lib64

# Set compiler flags and header files directories
CXXFLAGS = -Xcompiler="$(CPU_FLAGS) $(CPU_OPT) $(OPENMP) \
$(CPU_DEBUG) $(PROFILE) $(WARNING)"\
$(GPU_OPT) $(CPP_STD) $(GPU_DEBUG) \
$(GIT_HASH) \
$(INCLUDES) \
--device-c --restrict

# Set linker flags and library files directories
LDFLAGS = -Xcompiler="$(OPENMP)" \
-Xlinker="-rpath,$(HDF5_DIR)/lib:$(CUDA_DIR)/lib64" \
-std=c++11 \
$(LIB_PATHS)
endif

############################ NVCC + Intel icpc #################################
ifeq ($(COMPILER), Intel)
# C++ compiler for CUDA
CXX = /usr/pubsw/packages/CUDA/10.0/bin/nvcc

# C++ standard
CPP_STD = -std=c++11

# Enable OpenMP
OPENMP = -qopenmp

# Set CPU architecture
# Sandy Bridge, Ivy Bridge
ifeq ($(CPU_ARCH), AVX)
CPU_FLAGS = -m64 -xAVX

# Haswell, Broadwell
else ifeq ($(CPU_ARCH), AVX2)
CPU_FLAGS = -m64 -xCORE-AVX2

# Skylake-X, Ice Lake, Cannon Lake
else ifeq ($(CPU_ARCH), AVX512)
CPU_FLAGS = -m64 -xCORE-AVX512

# Maximum performance for this CPU
else
CPU_FLAGS = -m64 -xhost
endif

# Use maximum optimization
CPU_OPT = -Ofast
# Use maximum optimization
GPU_OPT = -O3

# CPU Debug flags
CPU_DEBUG =
# Debug flags
GPU_DEBUG =
# Profile flags
PROFILE =
# C++ warning flags
WARNING = -Wall

# Add include directories
INCLUDES = -I$(HDF5_DIR)/include -I.
# Add library directories
LIB_PATHS = -L$(HDF5_DIR)/lib -L$(CUDA_DIR)/lib64

# Set compiler flags and header files directories
CXXFLAGS = -Xcompiler="$(CPU_FLAGS) $(CPU_OPT) $(OPENMP) \
$(CPU_DEBUG) $(PROFILE) $(WARNING)" \
$(GPU_OPT) $(CPP_STD) $(GPU_DEBUG) \
$(GIT_HASH) \
$(INCLUDES) \
--device-c --restrict -ccbin=icpc

# Set linker flags and library files directories
ifneq ($(LINKING), DYNAMIC)
LDFLAGS = -Xcompiler="$(OPENMP) -static-intel -qopenmp-link=static"
else
LDFLAGS = -Xcompiler="$(OPENMP)"
endif

LDFLAGS += -std=c++11 -ccbin=icpc \
-Xlinker="-rpath,$(HDF5_DIR)/lib:$(CUDA_DIR)/lib64" \
$(LIB_PATHS)
endif

################################### Build ######################################
# Target binary name
TARGET = kspaceFirstOrder-CUDA

# Units to be compiled
DEPENDENCIES = main.o \
Containers/MatrixContainer.o \
Containers/CudaMatrixContainer.o \
Containers/OutputStreamContainer.o \
Hdf5/Hdf5File.o \
Hdf5/Hdf5FileHeader.o \
KSpaceSolver/KSpaceFirstOrderSolver.o \
KSpaceSolver/SolverCudaKernels.o \
Logger/Logger.o \
MatrixClasses/BaseFloatMatrix.o \
MatrixClasses/BaseIndexMatrix.o \
MatrixClasses/CufftComplexMatrix.o \
MatrixClasses/ComplexMatrix.o \
MatrixClasses/IndexMatrix.o \
MatrixClasses/RealMatrix.o \
MatrixClasses/TransposeCudaKernels.o \
OutputStreams/BaseOutputStream.o \
OutputStreams/IndexOutputStream.o \
OutputStreams/CuboidOutputStream.o \
OutputStreams/WholeDomainOutputStream.o \
OutputStreams/OutputStreamsCudaKernels.o \
Parameters/CommandLineParameters.o \
Parameters/Parameters.o \
Parameters/CudaParameters.o \
Parameters/CudaDeviceConstants.o

# Build target
all: $(TARGET)

# Link target
$(TARGET): $(DEPENDENCIES)
$(CXX) $(LDFLAGS) $(DEPENDENCIES) $(LDLIBS) -o $@

# Compile CPU units
%.o: %.cpp
$(CXX) $(CXXFLAGS) -o $@ -c $<

# Compile CUDA units
%.o: %.cu
$(CXX) $(CXXFLAGS) $(CUDA_ARCH) -o $@ -c $<

# Clean repository
.PHONY: clean
clean:
rm -f $(DEPENDENCIES) $(TARGET)
(END)

Posted 4 years ago #
Jiri Jaros

Developer
Joined: Feb '12
Posts: 115

Hi bastg,
the driver and compile parameters look good to me. Even, nvidia-smi says the GPU is in default mode thus anyone can use it.

could you try to run the code with -g 0 parameter. This will explicitly select GPU no 0.

Could you also try to run
./kspaceFirstOrder-CUDA --version -g 0

Best
Jiri

Posted 4 years ago #
bastg

Member
Joined: Jun '19
Posts: 7

Hi Jiri -- below is the output of the "--version -g 0" command line. Let me know if this gives you some insight... Thanks for your help!

>> !/autofs/space/guerin/USneuromod/KWAVE_1.2.1/src/kspaceFirstOrder-CUDA/kspaceFirstOrder-CUDA --version -i kwave3D_N104_650KHZ_0p75MM_PAR.h5 -o kwave3D_N104_650KHZ_0p75MM_SOL.h5 -g 0 ;
┌───────────────────────────────────────────────────────────────┐
│ kspaceFirstOrder-CUDA v1.3 │
├───────────────────────────────────────────────────────────────┤
│ Selected GPU device id: Failed │
├───────────────────────────────────────────────────────────────┤
│ Build information │
├───────────────────────────────────────────────────────────────┤
│ Build number: kspaceFirstOrder v3.6 │
│ Build date: Jun 3 2020 │
│ Build time: 12:25:49 │
│ Git hash: 468dc31c2842a7df5f2a07c3a13c16c9b0b2b770 │
├───────────────────────────────────────────────────────────────┤
│ Operating system: Linux x64 │
│ Compiler name: GNU C++ 7.3.1 20180303 (Red │
│ Processor name: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz │
│ Instruction set: Intel AVX 2 │
├───────────────────────────────────────────────────────────────┤
│ CUDA runtime: 10.1 │
│ CUDA driver: 10.1 │
│ CUDA code arch: 0.0 │
├───────────────────────────────────────────────────────────────┤
│ CUDA device id: 0 │
│ CUDA device name: N/A │
│ CUDA capability: 0.0 │
├───────────────────────────────────────────────────────────────┤
│ Contact email: jarosjir@fit.vutbr.cz │
│ Contact web: http://www.k-wave.org │
├───────────────────────────────────────────────────────────────┤
│ Copyright (C) 2011-2020 SC@FIT Research Group, BUT, Czech Rep │
└───────────────────────────────────────────────────────────────┘
┌───────────────────────────────────────────────────────────────┐
│ !!! K-Wave experienced a fatal error !!! │
├───────────────────────────────────────────────────────────────┤
│ Error: CUDA device id 0 is busy or unavailable. │
├───────────────────────────────────────────────────────────────┤
│ Execution terminated │
└───────────────────────────────────────────────────────────────┘
Segmentation fault (core dumped)
/autofs/space/guerin/USneuromod/KWAVE_1.2.1/src/kspaceFirstOrder-CUDA/kspaceFirstOrder-CUDA --version -i kwave3D_N104_650KHZ_0p75MM_PAR.h5 -o kwave3D_N104_650KHZ_0p75MM_SOL.h5 -g 0 ;: Segmentation fault
>>

Posted 4 years ago #

RSS feed for this topic

Reply

You must log in to post.

k-Wave

A MATLAB toolbox for the time-domain simulation of acoustic wave fields

Running on AWS Tesla instances

Reply

A MATLAB toolbox for the time-domain
simulation of acoustic wave fields