2016.09 Release Notes

New Features and Enhancements

Products Internal Number Case Number Description
All 6258   There is now a Python interface for the vovserver REST API. The Python interface returns a JSON formatted string as a result of posted query. For more information about request URLs and response format please refer to REST API documentation. As vovserver requires the user to be authenticated, the Python interface will automatically prompt the user for username and password if the current login session is expired.
Example scripts:
from common.scripts.rtda_rest import RTDARest
rtda_client = RTDARest()
obj = rtda_client.getUrl(&http://localhost:11913/api/v1/jobs/12345/status&)
print (&Return type is&, type(obj)) print (&Return val: &, obj)
All 2272   Selection rule syntax now supports several addtional features.

Logical "or" operations in selection rules are available via the | operator.

Logical operations may be grouped with parentheses.

AND operations always take precedence over OR operations in the absence of parentheses.

The words and, or and not can now be used in place of &, | and ! respectively; this makes shell-level scripting easier. These words are case-insensitive; they may also be typed as AND, OR and NOT.

The = operator is now supported, which is equivalent to the == operator.

Spaces are now allowed in non-quoted string values if they are preceded by a backslash '\' character. Example: command^sleep\ 60 is now a valid rule.
All 4218   Improved protocol for more compact packing.

To send 16-bit integer, packing size is changed from 8 bytes to 3 bytes. to send 32-bit integer, packing size is changed from 8 bytes to 5 bytes.

To send 64bit integer, packing size is changed from 16 bytes to 10 bytes.

By removing 4-byte alignment packing and reducing data type encoding method, encoding of string and double types is also more efficient.
All 4980   Daily log files generated by system daemons are now automatically compressed.
Note: This applies only to non-Windows platforms.
All 5349   Vovselect now supports the '*' wildcard to signify all fields of a particular object.
Note: Ensure the '*' character is quoted as required by your shell. Example: vovselect '*' from jobs where idint==12345
All 5595 13053 The increased web server security now prevents local file inclusion via a path relative to an open URL, such as /gif.
All 5935   Added waitreasons as from source to vovselect, which provides access to the data available in vovshow -waitreasons.
All 5954   Jobs submitted to NetworkComputer by the administrator can no longer be accidentally dispatched to the VOV database support slave.
All 5985   Added coresused and corestotal fields to vovselect when querying slaves.
All 6071   The interface to the vovversion shell command has been changed.
The new command interface:
% vovversion -h 
vovversion: Usage Message
 DESCRIPTION: 
 Print the VOV version. Optionally show more details.
USAGE:
 vovversion [options] 
OPTIONS: 
 -help -- Print this message 
 -all -- Print all available version information 
 -clients -- List clients and their version numbers 
 -install -- Print the VOV version with patch names, if any 
 -patches -- Print patch names, if any 
 -project -- Print the version of vovserver associated with the current project [Can abbreviate options provided they are unambiguous]
All 6107   Added timestamp for FILE type and start and end time for JOB job in vovconsole Navigator. This allows users to easily see the dependencies of a timestamp.
All 6119   For easier editing, Cut/Copy/Paste menus have been added to Text fields
All 6168   Added support for doing daily maintenance on the VOV database, which is configurable through the database administration web interface.
All 6196   A new configuration parameter, liverecorder.logsize, now controls the size of Live Recording log files.
The parameter, liverecorder.logsize, can be added to the policy.tcl file, or set on-demand for the server via:
 vovsh -x &vtk_server_config liverecorder.logsize N

or for the slave via: vovslavemgr configure -liverecorder.logsize N SLAVENAME where N is any number between 256 and 8192, which represents the size of the log file in megabytes. This setting must be applied before turning Live Recorder on.
All 6400   The protection against deleting a resource when a job is queued against is now improved; it no longer slows the vovresourced-based expiration extender.
LicenseMonitor 5394   Integrated licensing capabilities now support server lists. This feature is useful for unlicensed tool wrapping.
NetworkComputer 5766   New features have been added to the vovprocessmgr utility for finding processes that are descendents of vovslave, orphans of vovslave and external processes. There is an option to create foster jobs for discovered orphans; orphans can now be accounted for by a slave on the same host, and tracked for the rest of their lifetime.
NetworkComputer 5904   On Linux slaves, a job can now be requested to run in one or more cgroups.The syntax is similar to requesting any other resource, with the resource name consisting of the prefix CGROUP: followed by the path to the cgroup on the filesystem.
Example, to use /sys/fs/cgroup/cpuset/my_cgroup1 /sys/fs/cgroup/memory/my_cgroup2:
nc run -r CGROUP:/sys/fs/cgroup/cpuset/my_cgroup1 -r
                  CGROUP:/sys/fs/cgroup/memory/my_cgroup2 -- sleep 120

will assign a "sleep 120" job to the cgroups my_cgroup1 and my_cgroup2

If the user specifies multiple conflicting cgroups (such as 2 cgroups under the /memory hierarchy), the cgroup that is specified last is the one that the process will be assigned to.

The special resource CGROUP:RAM can be used to limit memory usage of a job within a cgroup.

Example:

nc run -r CGROUP:RAM -r RAM/2000 -- sleep 120 will assign the job to a default cgroup and limit that cgroup to 2000 megabytes of RAM. Since we only place one job in each default cgroup, we can effectively limit RAM usage on a per-job level. The path to this default cgroup will be: 
<path to cgroup root directory >/memory/ <queue name>_ <slave name>_ <job slot number>
Note: CGROUP:RAM cannot be used with a non-default cgroup; if both CGROUP:RAM and a non-default cgroup are specified, the job will be placed in the specified cgroup without changing that cgroup's RAM usage limit. We strongly recommend specifying a RAM resource when using CGROUP:RAM, as the default value is low (currently 20 megabytes).

To see slave resource for cgroups, use nc hosts -r and look for CGROUP: entries. If they are not present, ensure that they are set up on the slaves with lscgroups and check the slave logs for errors.
NetworkComputer 5909   The new dynamic server tuning feature enhances performance when the server is under heavy load conditions: a unique maximum value is set per bucket.
NetworkComputer 5080   Slaves now account for jobs running on a stopped slave on the same host. When a slave is started, if there is a matching slave in the stopped condition (waiting on its jobs to finish), the new slave will adopt any jobs on the stopped slave by using foster jobs. This feature helps prevent host overloading.
NetworkComputer 5472 12544 There is a new server configuration parameter for statistics: cpuprogressWindowSize.

This parameter can be set in the policy.tcl configuration file.

The default value (1) provides identical behavior to previous releases. Increasing the parameter increases the number of samples to be used in the calculation of the cpuprogress statistic.

The accepted range of values is 1-1440. 1440 signifies one full day, assuming the sample time is 60 seconds.
NetworkComputer 5689   The fields REQCORES, REQCPUS, REQPERCENT, REQRAM, REQSLOTS, and REQSWAP are now valid on SCHEDULED jobs. This can be useful in writing -preempting clauses on preemption rules.
NetworkComputer 6144 14414 Validation is improved with two new options and a new health check.

Summary:

The new -failover option to vtk_slave_define now sets the capacity to 0, which prevents the slave from accepting jobs and pulling a license. This also acts as a flag to perform some failover configuration testing, such as checking servercandidates.tcl to ensure the slave host is in the list.

Passing in the new -descriptors as well as the -failover option to vovslavestartup triggers a check to ensure the host has at least as many descriptors as vovserver, which ensures it can operate at full capacity in the event of failover. It also checks that the server_election directory is empty.

In addition, a new health check ensures that a slave is running on each host listed in the servercandidates.tcl file. By default, the health check is on; it can be disabled in the health check configuration of the web UI.
Note: The health check features and options are listed on the health check configuration of the web UI.
NetworkComputer 6195   Update calls to ftlm_lmproject from vw every 20m. The period is controlled by the variable VOV_VW_PING.
NetworkComputer 6204   The PERCENT slave resource can now be configured in the slave resource specification. This is specified in consumable form, such as PERCENT/50. Previously, PERCENT was always initialized to 100 and could not be changed.
  6312   New parameter resuserDisableMatchingThreshold allows matching to be disabled for license resources from a license server: in such cases, the matching process can take a very long time.

resuserDisableMatchingThreshold provides a threshold that disables matching if the sum of LicenseMonitor handles and FlowTracer jobs exceed that threshold.

The range is 0 - 10000; the default value is 1000.

This parameter can be specifed in policy.tcl.

Example:
set config(resuserDisableMatchingThreshold) 1000

This parameter can also be set using the VTK API.

Example:
vtk_server_config resuserDisableMatchingThreshold 1000
NetworkComputer 6722 14704 The vovserver now responds to multiple incoming client connections during the same cycle.
This capability is enabled by default, and can be controlled with the following congfiguration parameters via the policy.tcl file:
set config(clientConnectionQueue.mode) multi
set config(clientConnectionQueue.size) 512

The mode can be "single" or "multi" (default=multi). The size can be any integer in the range of 25-1024 (default=512).
Workload Accelerator 6558   The ncmgr and wxmgr utilities now check for a minimum number of file descriptors on Unix-based platforms.

Summary:

The default value can be over-riden with the VOV_MIN_DESCRIPTORS environment variable.

An attempt is made to raise the default limit if needed.

If the target's limit is higher than the system's hard limit, an error is displayed and the product start will be aborted.
LicenseAllocator 5311 12150 License resources can now be allocated from multiple servers or excluded.

Summary:

Resource groups can be defined that are a logical OR of components that are hosted on different license servers.

Licenses from specific servers can be excluded from distribution to specific sites.
FlowTracer 6358   A SNAPSHOT of the server environment is now automatically captured on FlowTracer vovproject start. This new feature enables users to run a job in the same environment as the server. A shell environment snapshot can also be saved on demand from any project enabled shell.

Summary:

The automatically captured environment is saved as a file under the server working directory, which can be used to set the SNAPSHOT environment at any time.

The new standalone utility vovenvcapture is for saving the environment on demand, either into a file or as a property on a Vov object.
FlowTracer 4879   A periodic job can be paused by setting a property "PERIOD_PAUSE" as 1. This is done through web interface, vovconsole NodeEditor, vtk_prop_set tcl function, or vovprop cli. If a paused job is running when this property is set, the current run will continue to completion, but a new run will not be started at the next period; it will remain paused until PERIOD_PAUSE property is reset to 0.
FlowTracer 5428   If EventOverflow occurs from server, vovconsole will empty all events, wait, reconnect and refresh all set viewers. When this occurs, a message is displayed: Updating ... please wait
FlowTracer 5617 13082 Upcone Set and Downcone set menus are now added to the popup menu of vovconsole when a set is selected. If no node is selected, Connectivity popup menus of Downcone, Upcone and Expand will apply to the current set displayed in the SetViewer.
FlowTracer 6018   Details about the condition of a schedule job that is stuck due to barrier invalid are now available.
Summary:
When a job is stuck due to barrier invalid node in upcone, a balloon message appears and shows "barrier invalid".
On NodeEditor, the "Why" tab shows barrier invalid messages on the associated node(s).
Trouble-shooting methods:
(1) An aggressive retrace flag can be done in the main console window or by using vsr.
(2) Find the invalid jobs in the upcone and retrace them to validate the barrier.
If there are no invalid jobs in the upcone, fix the barrier invalid flag by running sanity.
FlowTracer 6122   The Navigator column now sorts per the severity of the node status.
FlowTracer 6156   SERIAL and PARALLEL commands in FDL now work with any level of nesting: you can nest parallel in serial and vice versa, for as many levels deep as desired. SERIAL and PARALLEL commands now work with S (set), T (task), and J (job) commands, which can be nested under one another in any order.
FlowTracer 6440   vovlsfd now utilizes an agent script (vov_lsf_agent) in the installation to launch vovslave in the batch system. Previuosly, shell scripts had to be written to the launchers/<hourly sub-directory>.
FlowTracer 6571 15109 The vw command now has the option to apply a delay that allows latency on outputs as well as inputs.

Previously, if the environment variable VOV_VW_NFS_PROTECTION was set to 1 (or if the -n option is used with vw), vw added a delay when waiting for inputs to allow for Network File System (NFS) latency. By default, the wait duration was 60 seconds that could be overridden by setting the environment variable VOV_VW_NFS_WAIT.

In this release, in addition to the above behavior, if VOV_VW_NFS_PROTECTION is specified as 2 (or the -N option is used), vw also applies the same delay to allow for latency on outputs.

Overview:
Under certain conditions, for both inputs and outputs, vw will attempt to trigger a cache update sooner if possible. Note: for this to work, nfsdelay must be enabled in the vovserver settings in the policy.tcl file, and also applied to any currently running vovserver (by doing vovproject sanity). Example: set config(nfsdelay) 60
It is also recommended to specify a small time tolerance in policy.tcl to compensate for minor time discrepancies among machines and/or the filer. Example: set config(timeTolerance) 3
Note: This should also be applied to any currently running vovserver using vovproject sanity.

Resolved Issues

Products Internal Number Case Number Description
All 3120 13056 Increased web server security, which prevents cross-site scripting.
All 6132   Added arch field support for slave objects to VovQuery: vovselect, vtk_select.
All 6193   An issue has been fixed with field-only selection rules for string fields. By passing the field name without an operator or a value, you can now query for objects that have a non-empty value in the specified field. For integer fields, this form of selection rule queries for objects that have a non-zero value in the specified field.
All 6253   Scrolling has been added to the alerts dialog, which makes long lists of alerts easier to access and consumes less space on the monitor.
All 6394   Viewing the files of sets has been made easier:

Summary:

For default sets that only contain files, such as System:files, System:filesToCheck, System:zippable, Predefined:missing files, and Predefined:blocking files, files are always displayed regardless of the setting of the flag.

However, for user defined sets, Show files must be turned on to view the files.
LicenseMonitor 6103 13577 New format is now recognized, and added support for additional date format in Feature line.
LicenseMonitor 6268   Detailed plots are now reasier to read. The visibility of plot lines was increased for checkouts and queued requests with shorter durations compared to the report time range.
NetworkComputer 5253   The vovfsgroup create command now copies the parent ACL when creating a subgroup. Example: vovfsgroup create /abc/def will create a new group /abc/def, with ACL permissions copied from the group /abc. If there is no applicable parent group, the default ACLs will be used.
NetworkComputer 5699 14154 fairshare.cgi now takes fstokens into account. Previously, fstokens were ignored.
NetworkComputer 5715 13420 The autokill function now takes suspension time into account when determining if a job has exceeded the autokill time threshold.
NetworkComputer 5993 13705 License resources are no longer overbooked. Previously, the configured threshold was honored the first time the overbooking procedure was called for a specific feature.
NetworkComputer 6125 13164 vovset list can now display more than 600k sets.
NetworkComputer 6617   The failover process now works correctly.
Note: It can take over 2 minutes for the failover process to complete.
NetworkComputer 6618   The error message for a specific port issue is better handled: where ncmgr is being used to start or stop an NetworkComputer instance, but the specified port matches that of an NetworkComputer instance other than the one being controlled.
NetworkComputer 6619 15159 The DISPLAY environemnt is unset during the failover process, which prevents the process from failing due to an invalid value.
LicenseAllocator 6414   ADMIN permission is no longer needed to read LicenseMonitor or NetworkComputer data. That data is now accessible by all USERs. However, updating the NetworkComputer instances with new allocations still requires ADMIN privileges.
LicenseAllocator 6601 12150 The minimum number of tokens to run any queued job on a site are allocated when possible. This ensures jobs will be run.
LicenseAllocator 6752 15243 When a new resource is declared in the LA config.tcl file, restarting LicenseAllocator or vovlad is no longer needed to get the resource tokens.
LicenseAllocator 6753 15245 LicenseAllocator now uses the expression specified through LA::SetMapForResourceInSite to set the resource map expression in NetworkComputer; components are in the resource map in the correct order.
LicenseAllocator 6756 15244 LicenseAllocator now sets the weight of the components of a resource group from the weight of the resource group, unless the weight of the component has been set explicitly.
FlowTracer 4826 11903 When a new set is created in the Set Browser, the sets can now be placed in alphabetical order: right-click any node and select the Update and Sort option. Previously, new sets were appended to the bottom, with no option to update the order.
FlowTracer 4901 10461 vovproject enable now returns a non-zero exit when it fails.
FlowTracer 4945   Detailed information of "why" is now provided if a job fails with no outputs.
FlowTracer 5198   Improved Slave Monitor: Slave LED Monitor on vovconsole and Floating Slave Monitor.
Summary:
When there are many slaves, Slave Monitor will not occupy the vovconsole area and show only one row and rest of them can be seen at floating Slave Monitor
If there are more than 20 slaves, grouping options are displayed.
Users can group slaves by state, arch, group(=slavegroup)
FlowTracer 5231 3186 4562 5409 Predefined sets on the console can now be deleted or modified; they are not permanently deleted.
Summary of the changes:
After a predefined set is deleted, the vovconsole keeps the set names.
When the predefined sets are double-clicked, the predefined sets are recreated: the predefined set is recomputed and the up-to-date set is drawn on the set viewer.
FlowTracer 5424   Manually overriding vovslave cores and capacity is now improved.
Summary of features:
By default, the capacity follows the core count.
The capacity can be manually set via the -T option, or by defining the SLOTS/N consumable resource via the -r option. N is a positive integer.
In all cases, the capacity directly affects the number of slot licenses that will be requested.
FlowTracer 5537   Sticky attachment attributes on inputs are now preserved when creating job arrays. Previously, sticky attributes were not copied for inputs.
FlowTracer 6101 14221 vovconsole no longer crashes when running in READ-ONLY mode. Menus and buttons that are not applicable for read-only security are now disabled.
FlowTracer 6129 13269 A race condition that occurred between resource reservations and resource grabbing by a job has been fixed. Previously, this occurred when a job was dispatched by the server and resource was reserved between the time of the dispatch and the time the job started running on a slave.
FlowTracer 6262   The Disconnect popup menu of the Navigator is now fixed. Previously, error messages were displayed.
FlowTracer 6364   In vovconsole, jobs in a set are no longer deleted when the set is flattened then rebuilt from FDL.
FlowTracer 6438   The following commands that were previously deprecated are no longer available: vovstopjobs vovleader -K retraces
FlowTracer 6668   The numbers of projects for other users shown in registry.cgi page has been corrected.
FlowTracer 6683   Files no longer appear in sets that should not contain files such as "Predefined:stuff to do" and "System:jobs".