The SNAPSHOT and SNAPPROP Environments

The SNAPSHOT Environment

If the environment variable VOV_ENV is not defined or if it is defined and it contains the keyword SNAPSHOT, the submission procedure creates a snapshot file with all current environment variables, excluding some troublesome variables, listed later in this section. The snapshot file is typically in vnc_logs/snapshots/$LOGNAME/$VOVARCH/envNNNNN.env where NNNNN is a hash of the content of the file, and it is used to quickly share existing snapshots between files. After the snapshot file has been created, the environment of the job is modified so that the SNAPSHOT environment gets a parameter which is a full name or a relative name to the snapshot file. The snapshot file is chosen based on the value of the variables NC_LOGDIR and NC_SNAPSHOTDIR.

% nc run -e SNAPSHOT sleep 10
Fairshare= /time/users.john
Resources= macosx
Env      = SNAPSHOT(vnc_logs/snapshots/john/macosx/env13378.env)
Command  = vw sleep 10
Logfile  = vnc_logs/20121013/124703.55776
JobId    = 007470870

The SNAPPROP Environment

It has been proven that under heavy load, many NFS servers are not fast enough to deliver the SNAPSHOT file to the remote host on which jobs need to be executed. This is particularly tough to debug because by the time one is ready to investigate a job failure caused by a bad snapshot file, the file has become available on the remote host.

Accelerator implements an alternative way to deliver the snapshot information to the remote job which does not rely on NFS files but rather on properties attached to the job. The downside of this approach is an increase in memory use by the main vovserver in Accelerator, such on the order of one additional GB for about 100,000 jobs, although this does not seem to be much of a problem on current hardware.

% nc run -ep  sleep 10
Fairshare= /time/users.john
Resources= macosx
Env      = SNAPPROP(@JOBID@)
Command  = vw sleep 10
Logfile  = vnc_logs/20121013/123218.26892
JobId    = 007470866

% nc info !
Id,User,Group   007470866,john.staff,/time/users.john
Environment     SNAPPROP(007470866)
Directory       /Users/john
Command         sleep 10
Resources       macosx
Submitted from  mac09
Submitted at    Sat Oct 13 12:32:18 PDT 2012
Priorities      schedule=normal execution=normal
Status          Done
   Host         mac09
   Queue Wait   0s
   CPU Time     0.00
   Max RAM      0MB
   Duration     11
   Age          9s
   AutoForget   1

Customize SNAPSHOT Behavior

Some aspects of the environment snapshot procedure can be customized by the administrator via the $VOVDIR/local/vovenv.config.tcl file. The settings in this file apply to both file and property-based snapshots.

Variable Blacklisting

Other Settings

Example File

# Example of $VOVDIR/local/vovenv.config.tcl  file.
lappend badVarList SITE
set maxEnvSize 10000