k-Wave

1. mcdannold
   

   Member
   Joined: Sep '15
   Posts: 5

   Hello,

   I am running hundreds of simultaneous k-Wave simulations on a linux cluster. Each instance runs Matlab on a computing node and executes the simulation using compiled k-Wave code. After each simulation is run, it will run another one. So far, it's working fine. However, on occasion some of the instances are giving me trouble. It will crash when writing out the xxx_input.hdf5.

   It's not a huge deal, but any idea what might be going wrong? Perhaps something is not installed correctly on some of the nodes? I am running my script that runs the simulation using try/catch, so when I get the problem it moves on to the next simulation. When I get this error, it happens for all subsequent ones, which makes me wonder if it's something going on with the specific node.

   Here is what I see:

   Running k-Wave simulation...
   start time: 10-Dec-2021 17:31:03
   reference sound speed: 2000m/s
   dt: 48.7013ns, t_end: 133.7825us, time steps: 2748
   input grid size: 44 by 44 by 492 grid points (14.2857 by 14.2857 by 159.7403mm)
   maximum supported frequency: 2.31MHz
   expanding computational grid...
   computational grid size: 64 by 64 by 512 grid points
   precomputation completed in 0.83197s
   saving input files to disk...

   ME =

   MException with properties:

   identifier: 'MATLAB:imagesci:hdf5lib:libraryError'
   message: 'The HDF5 library encountered an error and produced the following stack trace information: ...'
   cause: {0×1 cell}
   stack: [8×1 struct]

   Posted 2 years ago #

2. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,643

   Assuming you're using kspaceFirstOrder3DC (or G), if you are creating files using lots of different processes, one possibility is collisions between the auto generated file name, which is unique only using a time stamp. For example, if process A and process B both try to create a file to the same scratch or home directory at the same time, this could cause collisions. You could avoid this by setting your own file name using the optional input 'DataName' and making sure it's unique.

   Posted 2 years ago #

3. mcdannold
   

   Member
   Joined: Sep '15
   Posts: 5

   Hi Brad,

   I don't think that's the problem. When I specify the scratch file names, in addition to adding the data and time as you suggest, I add a random integer to it to further reduce the odds for collisions. Also, once it starts happening it happens on all subsequent simulations called on that node.

   -N

   Posted 2 years ago #

4. Bradley Treeby
   

   Developer
   Joined: Mar '10
   Posts: 1,643

   That’s weird then. Is there any extra information in the stack trace in the exception you catch so we can see which call it gets tripped up on?

   Posted 2 years ago #

5. mcdannold
   

   Member
   Joined: Sep '15
   Posts: 5

   I'm starting a new batch today and will let you know

   Posted 2 years ago #

RSS feed for this topic

Reply

You must log in to post.