Integration with PBS

Accelerator Plus has added the capability to use PBS Professional as its base scheduler. For the 2019.01 release, jobs can be submitted through Accelerator Plus and run on a PBS complex.

This requires a PBS client/server to be installed on the host where Accelerator Plus is running (qsub and other commands must be accessible). The PBS hosts must have access to the queue's server working directory ($SWD).

In addition, you can connect with Accelerator using the PBS emulation interface. This should only be used for debugging purposes.

Complete the steps below to integrate with PBS Professional.

  1. Start a WX queue using the wxmgr start command.
  2. Configure the vovwxd daemon for PBS.
    1. Use the command wx cmd vovwxconnect -pbs.
    2. Tune your WX configuration by modifying parameters as needed in $SWD/vovwxd/config.tcl. Tunable parameters are described in that file. Some useful configuration parameters include: tasker,max, tasker,maxQueuedPerBucket and tasker,maxSubsPerBucket.
  3. Once the configuration is complete, you can submit jobs. Additionaly there are some PBS parameters that can be specified via job resources:
    • PBSdest:<destination queue>: Defines the destination of the job. The destination names a queue, a server, or a queue at a server.
    • PBSpriority:<priority> : Defines the priority of the job.
    • PBSproject:<project> : Allows a root user or manager to submit a job as another user.
    • PBSjobname:<name> : Declares a name for the job.
 wx run -r PBSdest:myPBSqueue PBSproject:myProjectName -- sleep 0

You can also use the vovwxconnect, with the pbsemul NCQ as a way to debug. 


vovwxconnect: Usage Message

DESCRIPTION:
    Utility to setup a connector for Accelerator Plus to a primary queue

OPTIONS:
    -h               -- This help.
    -v               -- Increase verbosity.
    -nc NCQ          -- Connect with specified Accelerator queue.
    -wx WXQ          -- Connect with specified Accelerator Plus queue
    -vovdir PATH     -- Specify VOVDIR for the Accelerator queue, if different
                        than that of Accelerator Plus.
    -ncconfigdir     -- Specify NC_CONFIG_DIR for the Accelerator queue, if
                        different than that of Accelerator Plus.
    -lsfemul NCQ     -- Connect with Accelerator but using the LSF emulation
                        interface (only for debugging)
    -dd              -- Connect with Direct Drive
    -pbs             -- Connect with PBS
    -pbsemul NCQ     -- Connect with Accelerator but using the PBS emulation
                        interface (only for debugging)
    -legacy          -- Connect using legacy daemons vovelasticd vovlsfd.
    -show            -- Show what is connected.
    -test            -- Validate connections by running a test job. The wx
                        queue is determined by the environment variable
                        VOV_PROJECT_NAME if set, otherwise by NC_QUEUE if 
                        set, or the default wx queue name otherwise.
    -nostart         -- Do not start the vovwxd daemon.
    -loglevel L      -- Set verbosity level for vovwxd with -nc and -dd 
                        only (0-6)

EXAMPLES:
   % vovproject enable MY_WX_PROJECT
   % vovwxconnect -nc vnc1 vnc2
   % vovwxconnect -nc vnc1 -nc vnc2
   % vovwxconnect -nc vnc_test -vovdir /some/path/to/a/vovdir/used/by/vnc_test
   % vovwxconnect -pbs
   % vovwxconnect -show
   % vovwxconnect -test
   % vovwxconnect -lsfemul vnc_test

Configure Accelerator Plus to Request PBS Resources

You can configure PBSResList to request PBS resources.

Use the steps below:

  1. Request PBS resources with the following: 
    wx run -r "PBSResList:select=1:ncpus=1" --sleep 0
  2. To request multiple resources, use the following: 
    wx run -r "PBSResList:select=2 PBSResList:resource1=1 PBSResList:resource2=abcd" --sleep 0