2019.01 Update 3 Release

New Features and Enhancements

The following new features and enhancements were introduced this software release:
Product Internal Number Case Number Description
All VOV-10550   Added inline documentation for the default vovcleanup config file, cleanup.config.tcl.
All VOV-6577   A new configuration parameter (liverecorder.logdir) has been added to allow the Live Recorder recording file directory to be specified. This can be used with both the vovservermgr configure and vovslavemgr configure utilities.

The default location for the server remains $SWD/../. The default location for the slave remains /tmp.

All VOV-9639, VOV-9640   Improved the functional relationship between the refresh operation and the reporting operations of the vovprocessmgr utility, improved the help text for all options that needed clarification, added warnings about timing and accuracy, and added host filter support to the refresh operation for orphans.
All VOV-9454 23741 Most of the VOV Tcl files from the installation package now contain a proper Altair copyright statement.
Accelerator Plus VOV-9862 24090 Fixed a bug that was causing waitreasons to be empty for buckets that are being updated but their jobs cannot be dispatched to slaves.
Accelerator Plus VOV-10019   Added new parameter in vovwxd /config.tcl CONFIG(limit,mode) that allows Accelerator Plus to properly handle the -limit option of wx run. Refer to config.tcl for more information.
Accelerator Plus VOV-10569 23888 Removed call to vovtcpkill from within the vovserver failover script.
FlowTracer VOV-5668 20732 We support the following new features for stopping and suspending jobs:

The -signals, -include and -exclude options to nc stop and vovstop -f are now functional.

The trace parameters defaultStopSignalCascade and defaultSuspendSignalCascade have been added to specify default behavior when requesting to stop or suspend a job, respectively. The trace parameters defaultStopSignalDelay has been added to specify default delay when requesting to stop a job, respectively.

The job property STOP_SIGNALS can now be used to set stop-signal cascades on a per-job basis.

The job property STOP_SIGNAL_DELAY can now be used to set the delay between stop signals on a per-job basis.

Accelerator VOV-10392   Added an SFD (single-file distributable) package for Accelerator. Also, the window for all SFDs can no longer be resized.
Accelerator VOV-9596 24133 Added -json and jsondoc arguments to vsy to return job information in JSON format.
Accelerator VOV-9909   Added an option (-splitstderr) to write stderr output of the job to the terminal's stderr when running interactive jobs (nc run -I). The default is to write the job's stderr output to the terminal's stdout. Note that using this option will probably result in garbled terminal output due to interleaving of stdout and stderr outputs.

This feature requires an update of the vovsh, vovslave and vovserver binaries.

Accelerator VOV-10488   Added new STATISTICS field to the SET object that contains the break-down of jobs in a set, per job state, in JSON format.
Accelerator VOV-10489   Added new fields to SERVER object:
  • INNERLOOPTIMERS - json object containing percent of time spent in various phases within the server inner loop
  • NUMBUCKETS - total number of buckets
  • SERVERSIZE - the ram size of the server data
  • SCHEDMAXEFFORT - current setting of the schedMaxEffort trace parameter
  • CYCLEFREQ - avg server cycles/sec over the previous ~10s interval
Accelerator VOV-8857 22729 Added new slave resource: OSNAME. OSNAME is the name of the OS vendor/distribution (CentOS, Ubuntu, Windows, MacOS. etc).

Changed the value for the OSVERSION resource to be more useful. OSVERSION is the common version number associated with the release, such as 18.10, 6-2.

OSVERSION can be compared with =, !=, >, <, >=, and <= . A shorthand version such as 18, will match all 18.x.y versions. 18.4 will match all 18.4.y versions.

"nc run -r osname=Ubuntu 'osversion>=6.2' 'osversion<=7.6' -- command" will run on a slave running Ubuntu versions 6.2 - 7.6 inclusive.

Accelerator VOV-10223   Added wording to lmmgr usage help detailing how to disable the guest access port.
Accelerator VOV-8925 22887 Jobs placed under the spread policy will now favor the socket with the most free memory when the cores used count is equal between all sockets. Additionally, the NUMA_LAYOUT slave property has been augmented to show the memory currently used by all jobs on the slave that requested NUMA support (e.g. "Socket: 0 RAM= 500/32089 **oooooooooo").
Accelerator VOV-9021 23089 Allow for VovUserError to be called from within a job class as a means of exiting the job class without affecting the job class web UI page or the nc jobclass command.
Accelerator VOV-4897 20586, 21030 FairShare configuration page now permits direct weight entry without resetting the window to 0.
Accelerator VOV-9364 23619 Users can now specify the NUMA-related resources MAXNUMACORES and MAXNUMACORESFREE when running jobs. The former refers to the maximum number of NUMA cores in a single node on the slave, the latter to the maximum number of free NUMA cores on any single node on the slave.
For example: 
nc run -jpp pack -r+ MAXNUMACORESFREE#4
will request a job on any slave with at least 4 free NUMA cores on the same node.

In addition, the fields MAXNUMACORES and MAXNUMACORESFREE have been added to the SLAVES object for the purposes of vtk_select_loop, vovselect and related commands.

Accelerator VOV-10363 24474 Fixed requested RAM value shown by ram_usage.cgi
Accelerator VOV-10540   Licenses are now checked in upon ncmgr stop -freeze.
Accelerator VOV-10478   Metrics are now enabled by default. Added new scheduler metrics:
  • scheduler/cores/capacity
  • scheduler/cores/inuse
  • scheduler/hosts/ready
  • scheduler/hosts/full
  • scheduler/hosts/overloaded
  • scheduler/hosts/sick
  • scheduler/hosts/noslot
Internally calculate moving averages for the above metrics as well as all existing metrics:
  • scheduler/ram/capacity
  • scheduler/ram/inuse
  • scheduler/slots/capacity
  • scheduler/slots/inuse

Added new CAPACITYMETRICS field to the SERVER object that contains the latest value and moving averages for these metrics in json format.
Accelerator VOV-7408 20586 Original: FairShare configure page now displays actions for a group based on user's permission on that group.

This fix: Randomly, the fairshare.cgi web UI page would receive an error from the API that prevented it from rendering.

Accelerator VOV-9647 24675 Simplified the controls and behavior of the limit functionality of the tabular reports found in all products.

For reports that deal with a potentially high number of records, the separate "records to retrieve" control has been removed and consolidated into the table's built-in limit control. In such tables, the table no longer attempts to show the total number of rows available, nor the number of rows that were matched by the back-end query. This is done for both performance and accuracy reasons, as getting these numbers can take a significant amount of time from the back-end database.

Also fixed an issue that caused the "show all rows" link to not appear in some web UI tables that were being limited.

The behavior of this link has also changed slightly, in that when showing all rows, instead of passing the total number of rows the table is cognizant of back into the table as the new limit, a limit of 0 is passed. This triggers the table functionality to show all rows available. To prevent confusion, an empty string is displayed as the current limit value instead of 0. Before this change, the total number of rows was only current at the time the table was build, so the limit passed was often stale and still did not show all rows.

Accelerator VOV-10531   The "default" keyword is now supported for the log file specification for UNIX-based slaves that are managed by vovssd. This results in a composed log file specification of @SWD@/logs/slaves/<slaveName>/<slaveName> that will be passed into the vovslave -l option.
Accelerator VOV-9593 21346, 23449 The NC integration code will now load correctly with versions of Cadence ADE environment that have alphanumeric characters in their version strings (for example, "ICADVM18.1-64b").
Accelerator VOV-9779 23926 The vovcleanup command has been enhanced to remove old logfiles generated by vovwxd. The guiding age threshold is controlled by the new vonfig(cleanup,vovwxd) setting in the cleanup.config.tcl configuration file. Default value is 30d. This value can be changed by modifying (or adding, as necessary) the value in the cleanup.config.tcl file in the project's server working directory. For example set config(cleanup,vovwxd) 15d.
Accelerator VOV-6173 21118 Reservation by negation for vovslave is now accepted. For example:
nc cmd vovslavemgr reserve -jobclass \!normal -duration 2h a003

Resolved Issues

The following issues were resolved in this software release:
Product Internal Number Case Number Description
All   24362 Fixed a bug when vovserver incorrectly reports about license expiration for perpetual license.
All   23689 Added a new time value trace parameter: preemption.rule.cooldown. Preemption rules that are disabled due to exceeding the max preemption processing time (preemption.max.time.rule) will observe a cooldown period equal to the value specified by preemption.rule.cooldown before being automatically re-enabled. The minimum value is 0 which indicates that an alert should be generated but the rule will not be disabled. The default and max value is 1y = off (rules will not be automatically re-enabled).
All VOV-10317   Fixed an error that corrupted non-default values for the allowUidForSecurityFile parameter, which in turn could prevent the vovserver from honoring settings in the security file.
All VOV-10665   Fixed an issue in the deprecated vtk_resourcemap_reserve Tcl command that caused an error when attempting to cancel existing reservations by updating the duration to 0 seconds.
All VOV-10239   Simplified switches to avoid user confusion.
All VOV-10726   Fixed an issue that caused the "show all rows" link to not appear in some web UI tables that were being limited. The behavior of this link has also changed slightly, in that when showing all rows, instead of passing the total number of rows the table is cognizant of back into the table as the new limit, a limit of 0 is passed. This triggers the table functionality to show all rows available. To prevent confusion, an empty string is displayed as the current limit value instead of 0. Before this change, the total number of rows was only current at the time the table was build, so the limit passed was often stale and still did not show all rows.
Accelerator Plus VOV-10462 24488 Fixed a bug in vovwxd daemon that was causing it to request only 1 slave at a time for jobs with NC:<queue> resource.
Accelerator Plus VOV-10530 24543 Fixed a bug in vovwxd that was causing it to create only 1 launcher job per buckets processing cycle. The impact of the bug was slow Accelerator Plus job starts due to delays in launching the host job in the base NC queue.
Accelerator Plus VOV-10414 24497 Fixed a bug that was causing vovwxd daemon to run into a infinite loop.
Accelerator Plus VOV-10678 24572 Fixed an issue that caused user-based security rules to eventually stop working on a heavily loaded server.
Accelerator Plus VOV-10205   In Accelerator Plus, a Tcl error occurred in vovnc driver script, for one bucket, will not impact the processing of other buckets. However the error will be reported as an alert, the user should take action accordingly. The failed bucket will be recovered after vovnc.tcl file is updated.
Accelerator Plus VOV-10182 24321 In Accelerator Plus, user will not be able to submit jobs into a locked FairShare group. The FS group must be also locked on Accelerator Plus side as well as on Accelerator queue.
Accelerator Plus VOV-10355, VOV-10353   Job data files have been enabled for Accelerator Plus.
Accelerator Plus VOV-10161   vovresourced is disabled in Accelerator Plus.
Accelerator Plus VOV-9728 24019 wx run can now specify -G parameter for FairShare group. The wxagent job in the base queue will run with the specified group.
Accelerator Plus VOV-10275 24422 Removed syntax error warning from the slave log file
Allocator VOV-9066 23158 The "show all rows" links in LA resource report tables has been restored.
Allocator VOV-10841 24713, 24756 Fixed issue where Allocator would double count OOQ tokens. This would result in sites with OOQ jobs getting more than their fair share of tokens. This could also result in some tokens not being allocated to sites.
FlowTracer VOV-10187   Client connections made by scripts run by the auto-start feature now close correctly when the script exits, instead of remaining open indefinitely.
FlowTracer VOV-10658 24604 Fixed keyword expansion in log file names for array jobs and added support for cases where the array reference job ID is the intended target for an artificial dependency (AD) as opposed to each individual job in the array. This is accomplished via a new format for the AD target job: ARRAY:N, where N is the job ID of the array reference job.
FlowTracer VOV-10581 24030 Running commands via nc cmd ... or wx cmd ... will now apply the appropriate vnc_policy to the command's environment
FlowTracer VOV-10337   vovconsole hang issue has been fixed by handling duplicate IDs.
FlowTracer VOV-10901   Fixed an issue where an undefined variable was being referenced upon an LSF job submission failure in vovlsfd.
Accelerator VOV-10119   A low swap space alert will no longer be generated for servers with 0 total swap or total swap that is lower than the threshold.
Accelerator VOV-10403   A separate Windows user name is no longer required for user impersonation in Windows slaves. It is still required, however, if the Windows user name is different from the user name obtained from the job submission, as might be the case in a cross-platform submission environment.
Accelerator VOV-9883   AAP command line tools no longer throw errors when the output is piped through another command line program that exits before allowing the input command to finish, such as the UNIX "head" command.
Accelerator VOV-10072   An alert will be generated if all jobs on a host are stuck (consuming no CPU time). Optional alerts are available for setting a threshold for creating alerts based on a number of jobs being stuck per host, tool, or user. See the Jobstuck Procedure of vovnotifyd.
Accelerator VOV-10142 24314, 24325 An alert will now be generated if the process spawned to save the PR file fails.
Accelerator VOV-9482   Enhanced stripTclComments filter to retain repository IDs
Accelerator VOV-10784 24660 Fixed an issue with vovreconciled not revoking sum resources when the component resource that was revoked contains zeros in its name.
Accelerator VOV-10748   Fixed an undesired growth in memory consumption for jobs that utilize license-based resources.
Accelerator VOV-10401   Fixed issue that prevented the vovauxpasswd utility working in the Windows command prompt.
Accelerator VOV-10412   Fixed numerous issues regarding user impersonation on Windows. Jobs should now execute correctly as the target user on slave and job output should be available in the log file.
Accelerator VOV-10217   Fixed an issue with "default" slavelist.
Accelerator VOV-10410   Fixed issue with SNAPSHOT environment that caused UNIX-style path delimiters (colons) to be used on Windows instead of semi-colons for the VOVDIR/bat, VOVDIR/bin, and VOVDIR/scripts segments of the PATH environment variable.
Accelerator VOV-10781 24680 Fixes a crash related to nc info, vtk_explain_status and related commands when querying a job with pending dependencies (e.g. if the job was run with nc run -dep).
Accelerator VOV-7570, VOV-9885 20881, 24152 If $NC_LOGDIR is a non-existent directory, nc run failed in prior releases. This change causes nc run to create the $NC_LOGDIR directory if it does not exist and successfully launch the job.
Accelerator VOV-10616 24589 In jobclass editor, the autokill field is now free-form.
Accelerator VOV-10914 20586 Randomly, the fairshare.cgi web UI page would receive an error from the API that prevented it from rendering
Accelerator VOV-8980, VOV-9841 22986 Removed appearance of DOS newline characters (^M) when the output of nc run -I is piped.
Accelerator VOV-9877   Show the Accelerator-specific job status (eg Running) instead of the FlowTracer-specific status (RETRACING) in the UIs for all products other than FlowTracer.
Accelerator VOV-10356   Stderr will now be included in the output log for default windows jobs.
Accelerator VOV-9619   The network port opened on the submission host when running an interactive job now authenticates the incoming connection to prevent accidental connections, such as those made by port scanners, from causing the interactive job to fail.
Accelerator VOV-10408   The user profile directory is now created upon starting a job on a Windows slave if the directory does not already exist on the slave host.
Accelerator VOV-10107   Updated minMemFreeOnServer and minSwapFreeOnServer configuration parameter descriptions with the unit of measure, which is MB.
Accelerator VOV-10792 24689 Fixed issue with vovresourced failing to start when vtk_flexlm_monitor -tags { } is used with more than one tag.
Accelerator VOV-10831   A couple of issues with running Accelerator in interactive mode (nc run -I) have been fixed. The first issue would occasionally result in the initial output of the job being lost. The second issue would, in rare cases, cause interactive mode to time out and fail.
Accelerator VOV-10989 24514 Post execution script handling is changed for the case where a job body script terminates due to a signal or nc stop. In these cases, the post-execution script specified by -post on the nc run command will run to completion. A second signal or nc stop executed while the post-execution script runs can be used to terminate it early if that is desired.
Monitor VOV-10265 24389, 24416, 24487 Fixed an issue in the license web UI page that caused an error to be displayed concerning an undefined "enterpriselicense" array element when using a license key file for product licensing.
Monitor VOV-10013 23315 Parser now can handle datetime strings that has comma after year, such as "8/5/19, 10:14:06 AM" or "Jan 15, 2019, 5:27:40 PM"), and added test cases; Also fixed the parser to locate and use full key file path (necessary for commandline run).
Monitor VOV-10132   More places changed to use global db handle to prevent or essentially reduce multiple db handles in the same session. Also optimized code to reduce the repeated calls to db to retrieve the same data.
Monitor VOV-10049 24281 This addresses an issue where under certain circumstances the LM/LA vovlmd daemon crashes because the vov slaves are unable to stop jobs via the autokill mechanism.
Monitor VOV-10725 24614 More places changed to use global db handle to prevent or essentially reduce multiple db handles in the same session. Also optimized code to reduce the repeated calls to db to retrieve the same data.

Errata

The following issues and defects are known to exist in 2019.01 Update 3 software release.
Product Interal Issue Description
Monitor VOV-10947 Altair Monitor SFD not starting on Linux