Distributed Parallel Support

The active partial job, which starts the execution upon completion of dispatching of all partial jobs, is normally component number 1. This can be overridden by using the dp active argument.

% nc run -dp 8 -dpactive 8 uname -a

In the following example, the first partial job is dispatched to a Linux machine, the second partial job to a macosx machine, and all other partial jobs go to UNIX machines. The active partial job is number 2, which is the one running on macosx. The tasker that is running UNIX would becomes the master for the Distributed Parallel job set. This also works with tasker names, or any other tasker resource for that matter (kernel, RAM, etc.).

% nc run -dp 10 -dpres "linux,macosx,unix " -dpactive 2 vovparallel clone mywrapper

When a Distributed Parallel job is submitted, the partialTool job sets the following properties on the top-level job:

DP_ACTIVERANK

The rank of the active partial job, which is the partial job that launches the top-level job.

DP_ATTEMPTS

The number of times the partial jobs have failed to be allocated within the allowed time.

DP_COHORTWAIT

This property is created when -nocohortwait is passed with nc run. This instructs partialTool for each cohort task to finish when its subtask process has finished rather than wait for the primary job to complete (which is the default behavior). Using -nocohortwait sets DP_COHORTWAIT to zero (0). Otherwise, the default is 1.

DP_FAILURE

Explanation of failures.

DP_HOSTS

List of hosts for the distributed parallel job . This property is set when the last partial job is launched. The list may contain duplicates.

DP_PORT_X

This is set by partial tool X to the pair host dp_port.

DP_SEMAPHORE

Used by partial jobs to count the rank.

DP_SEMAPHORE2

A second synchronization counter used by partialTool.

DP_SETID

The ID of the set that contains the list of partial jobs.

DP_WAIT

Tells the partial job how long to wait before giving up the slot (default 30 sec).

DP_WAIT_MAX

This is the maximum allowed value for DP_WAIT for a job (default 30 mins).

In addition to these properties on the top-level job, the following property is set on all other jobs:

DP_PART_OF

The top-level job ID.

For example, two jobclass definitions mycalibre_a.tcl and mycalibre_b.tcl are defined as follows:

::::::::::::::
mycalibre_a.tcl
::::::::::::::
# Copyright (c) 1995-2021, Altair Engineering
# All Rights Reserved.

# $Id: //vov/trunk/src/scripts/jobclass/short.tcl#3 $


set classDescription "My Calibre job resources"

puts "This is mycalibre jobclass"
if { [ info exists VOV_JOB_DESC(dp,resources) ] } { 
  set VOV_JOB_DESC(dp,resources) [ string cat $VOV_JOB_DESC(dp,resources) ",RAM/200 CORES/2" ]
  set VOV_JOB_DESC(dp,jobclasses) [ string cat $VOV_JOB_DESC(dp,jobclasses) ",mycalibre_a" ]
} else { 
  set VOV_JOB_DESC(dp,resources) "RAM/200 CORES/2"
  set VOV_JOB_DESC(dp,jobclasses) "mycalibre_a"
}

proc initJobClass {} {
}
::::::::::::::
mycalibre_b.tcl
::::::::::::::
# Copyright (c) 1995-2021, Altair Engineering
# All Rights Reserved.

# $Id: //vov/trunk/src/scripts/jobclass/short.tcl#3 $

set classDescription "My Calibre job resources"

puts "This is mycalibre2.tcl"
if { [info exists VOV_JOB_DESC(dp,res,*)] }  { 
  puts "VOV_JOB_DESC(dp,res,*) already exists.  This is good!" 
} else { 
  puts "VOV_JOB_DESC(dp,res,*) does not appear to exist.  This is bad."
}

if { [ info exists VOV_JOB_DESC(dp,resources) ] } { 
  set VOV_JOB_DESC(dp,resources) [ string cat $VOV_JOB_DESC(dp,resources) ",RAM/400 CORES/4" ]
  set VOV_JOB_DESC(dp,jobclasses) [ string cat $VOV_JOB_DESC(dp,jobclasses) ",mycalibre_b" ]

} else { 
  set VOV_JOB_DESC(dp,resources) "RAM/400 CORES/4"
  set VOV_JOB_DESC(dp,jobclasses) "mycalibre_b"
}

proc initJobClass {} {
}

You then call nc run as:

nc run -v 5 -C mycalibre -C mycalibre2 -e BASE -J jeffjob -dp 4 vovparallel clone sleep 1

Regardless of the specified wait time, there is a maximum wait time (default 30 mins) that can be specified by manually setting the Distributed Parallel_WAIT_MAX property on the job. Use the P option with nc run, such as

-P DP_WAIT_MAX=12.0

An example of a script used with vovparallel clone is available in $VOVDIR/training/vnc/simple_dp_script.csh.

#!/bin/csh -f
#
# Example of a script to be used with  vovparallel clone.
# 
# Example of usage:
# % nc run -dp 4 simple_dp_script.csh
#

set sleepTime = 30
if ( $#argv > 1 ) then
   set sleepTime = $1
endif

# Each application has its own way to determine a rendezvous port.
# In this example, it is a fixed port.
set APPLICATION_PORT = 2345

if ( ! $?VOV_DP_RANK ) then
    echo "ERROR: Variable VOV_DP_RANK not defined."
    echo "       This script needs to be run with vovparallel clone ..."
    exit 0
endif

echo "Hello! I am component $VOV_DP_RANK"

if ( $VOV_DP_RANK == 1 ) then
    echo "This is the master. "
    echo "This should open a socket for communication e.g. $APPLICATION_PORT"
    set DP_HOSTS      = `vovprop GET $VOV_DP_TOPJOBID DP_HOSTS`
    echo $DP_HOSTS
    sleep $sleepTime
else 
    echo "This is the tasker"
    set DP_HOSTS      = `vovprop GET $VOV_DP_TOPJOBID DP_HOSTS`
    set masterHost = $DP_HOSTS[1]
    echo "This should communicate with master through ${masterHost}:$APPLICATION_PORT"
    sleep $sleepTime

endif

exit 0

For example, if you want the application to run with N components (possibly on different hosts), you can submit it with:

% nc run -dp <N>  vovmpirun ./path/to/application