k-Wave

1. DLam
   

   Member
   Joined: Oct '16
   Posts: 19

   Dear K-wave team,

   Our team has been running the K-wave C++ binary on a Linux cluster, submitting jobs within a bash script. It works great if we ensure that all workers are on a single node -- when the job is assigned to workers across multiple nodes, I notice in the console output, the 'Number of CPU Threads' heading shows a smaller number than the ones assigned when submitting the job.

   We were wondering if this is simply a display bug, or if something else is going on? We've been looking at increasing our worker assignments but the 'Time to Go' estimation doesn't change much when we double core count.

   Cluster:
   OS: Linux CentOS 7
   Cluster architecture: Multiple hosts each with multiple workers, running openMPI
   Job submission system: LSF

   Posted 5 years ago #

2. Jiri Jaros
   

   Developer
   Joined: Feb '12
   Posts: 115

   Hi DLam,

   the default behavior is to use as many cores as the system has. If it helps, I call routine omp_get_num_procs() to find out this number.

   I guess the cluster schedules sets the variable OMP_NUM_THREADS to inform OpenMP codes about how many threads to use. This is not implemented directly because it does not reliably work on Windows.

   You can to use -t parameter to set up the number of threads manually, e.g.
   kspaceFirstOrder3D -t ${OMP_NUM_THREADS}

   If you want to check, the cluster scheduler sets up the OpenMP environment correctly try:
   OMP_DISPLAY_ENV=true kspaceFirstOrder3DC -t XXX

   Reading the post again, you may ask about something else. Do you want to run the code across multiple nodes? If so, this is not possible. It requires a completely different implementation using message passing. If this is the case, send me an email.

   Best wishes
   Jiri

   Posted 5 years ago #

3. DLam
   

   Member
   Joined: Oct '16
   Posts: 19

   Dear Jiri,

   Thanks for the quick reply!
   Yes, we want to run the code across multiple host servers (each server with multiple sockets for CPUs, each CPU with multiple cores). If I read you correctly, the current C++ binary is incompatible with doing this message-passing between different servers?

   Best regards,

   Daniel

   Posted 5 years ago #

4. Jiri Jaros
   

   Developer
   Joined: Feb '12
   Posts: 115

   Hi DLam,
   yes, the released codes are targeted at multi-socket multi-core systems, but with shared memory (NUMA nodes).

   There is a distributed version, that can run across the whole cluster using OpenMPI. It is a release candidate 1 version. I plan to release it as soon as I've finished the current k-Wave release with 2D and axisymmetric codes for NUMA nodes. (I had to put it aside for two months as this releases has the priority).

   If you want to test the prototype, send me an email (jarosjir[ at ]fit.vutbr.cz) and I could become a beta tester.

   Best
   Jiri

   Posted 5 years ago #

5. ahmedsallam
   

   Member
   Joined: Sep '21
   Posts: 6

   Hello Jiri,

   Can you also please send me the OpenMPI code? I have sent you an email.

   Thanks,

   Posted 2 years ago #

RSS feed for this topic

Reply

You must log in to post.