Troubleshooting

Before troubleshooting Monitor, it is recommended that you make yourself familiar with its theory of operation.

Components

Monitor is comprised of components that perform various functions. Refer to the table below.

Component Description
vovserver Controls all tasks. Receives and manages checkout data. Serves web pages.
vovtasker The tasker (worker) that runs jobs to sample the license daemons and parse the resulting information.
vovlmd Daemon responsible for setting up monitoring on configured license servers. vovlmd controls the processes that run to create sample data for each tag.
vovnotifyd Daemon responsible for sending e-mails based on events that occur in the system or the licenses being monitored
vovtriggerd Daemon responsible for archiving parsing job logs upon job failure. This daemon should remain off unless this feature is required to debug an intermittent monitoring failure. See the hints section below for more details.
vovnginxd Daemon responsible for handling secure client connections and forwarding traffic to the vovserver. In Monitor, this daemon is on by default.

Daemon Control

The auxiliary daemons are normally launched at vovserver startup by the scripts in the licmon.swd/autostart directory. Check this if the auxiliary daemons are not starting properly. Example scripts are located in $VOVDIR/etc/autostart.

Daemon status can be checked and controlled, by visiting the Daemons page under the Admin tab, or by using the vovdaemonmgr command at the shell. The following table shows its usage information.


vovdaemonmgr: Usage Message

This is the command to show the status of daemons. You can also use this
command to start/stop the daemons.

NOTE: This command can only be used by the owner of the vovserver and on the
machine where the vovserver runs.

USAGE:
    vovdaemonmgr <SUBCOMMAND> [OPTIONS] [daemonsList]

SUBCOMMAND is one of:
    list     -- List the configured daemons. (list -all for all daemons)
    restart  -- Restart the specified daemons.
    show     -- Show the status of the specified daemons.
    status   -- Same as 'show'.
    start    -- Start the specified daemons.
    stop     -- Stop the specified daemons.

[daemonsList] is optional. When omitted, act on all daemons.

OPTIONS:
    -f         -- Force flag (for start only). If the start subcommand
                  has an explicit daemonsList, the specified daemons
                  will be started even if not configured.
    -h         -- Help usage message.
    -v         -- Increase verbosity.
    -retry <N> -- For NIS, control how many retries to attempt, default 0
    -wait  <N> -- For NIS, control how long to wait between retries, default 0s

EXAMPLES:

    % vovdaemonmgr list
    % vovdaemonmgr list -all
    % vovdaemonmgr status
    % vovdaemonmgr status vovpreemptd
    % vovdaemonmgr start vovnotifyd
    % vovdaemonmgr start -force vovresourced
    % vovdaemonmgr start -force -v -v vovresourced
    % vovdaemonmgr stop vovlad
The main operations supported by the vovdaemonmgr command are:
  • status - show daemon status
  • stop - take down one or more daemons
  • start - begin running one or more daemons

To determine the current situation, we recommend that you begin with status then use stop/start as needed.

Troubleshooting Hints

Below are some possible problems and items to check if you encounter them.

Problem Check
The web interface is not responding
  • The vovserver is down.
  • The vovnginxd daemon is down.
  • DNS is not configured to allow a route to the Monitor server host.
  • A firewall may be running that is blocking access to the Monitor server port (5555 by default).
  • If the Altair Accelerator installation or the licmon.swd is located on a network share, verify that the share can be reached from the Monitor server host.
Not receiving updated data from license servers
  • The vovlmd daemon must be running to schedule the jobs. Check it via the Admin > Daemons page.
  • Check that the parser tasker is running via the Tasker page page.
  • Check that the load average on the system is not too high. The default parser tasker has 4 job slots and a maxload of 10.0. If the load average is too high, the parser will halt the execution of new jobs until the load average drops below the maxload value. Try to prevent other CPU-intensive processes from running on the same machine.
  • Check the output of the most recent parsing job via the Current > Raw Data page, or in the licmon.swd/vovlmd directory to see if the status command is having problems executing.
Alerts for a license server being monitored are intermittently thrown Status command failures are commonly seen, possibly due to network connectivity glitches, bugs in the license server status command, or other external influences. Since Monitor keeps the most recent parsing job output only by default, this log cannot be used to track down the root cause of intermittent failures. There is a advanced debug feature available to keep an archive of all parsing runs for these cases though. To enable this feature, visit the Admin > Daemons page and activate the vovtriggerd daemon. This daemon will copy (and on UNIX, compress) each parsing job log if the job fails. The copies are stored in the licmon.swd/logs/parser directory. Only use this feature when needed, since it can consume 2-3x the disk space required for normal operation.
No historical data is being displayed
  • If the installation is new, it will take at least one hour for the database to be loaded with data by default.
  • If viewing either the Daily Statistics or Daily Plots page under the History > Features page, the summarized data for these reports are built overnight, so do not expect to see data for the current day.
  • Make sure that the product edition is set to Monitor and not Monitor-basic. This edition provides real-time data to the Accelerator product and does not provide historical capability. The product edition can be configured via the Admin > System page.
  • Verify that the license is visible and valid via the Admin > License page.
  • Verify that the live_load_checkouts task script exists in the licmon.swd/tasks directory.
  • Verify that the vovdbd daemon is running via the Admin > Daemons page.
  • Verify that the database is running via the Admin > System > Database Information page.
Images are not showing up in web pages The images are retrieved from the readonly port (5556 by default) of the licmon server via HTTP.
  • Check that readonly port is enabled.
  • Check DNS/NIS setup and /etc/nsswitch.conf to be sure that the licmon hostname resolves.
  • Set env-var VOV_HOST_NAME in licmon.swd/setup.tcl to a value that resolves to the LM server machine.
  • If an IP address is the only way to access the machine from other hosts, add VOV_HOST_HTTP_NAME with the desired IP address to the setup.tcl file and perform a reread.
    To make the change effective immediately, also enter the following commands at the shell. This sets the env-var in the running vovserver.
    % vovproject enable licmon
% vovsh -x "vtk_server_setenv VOV_HOST_HTTP_NAME <VALUE>
    If you are viewing Monitor over a port-forwarded tunnel through ssh, e.g -L 5555:jaguar:5555, the host names differ on each end of the connection. The only way we know to deal with this is to make the Monitor host an alternate name for 'localhost'. For the above example, where the remote host is 'jaguar', your line in the hosts file would be similar to:
    127.0.0.1 localhost jaguar
Drop-down menus do not stay open in Internet Explorer This is caused by compatibility view, which forces IE to render pages and process scripts in an older engine that is not always compatible with newer web technologies. Turn compatibility view off to ensure navigation and plotting elements work correctly.
Not receiving notification e-mails
  • The vovnotifyd daemon is not configured or is down.
  • The SMTP configuration may not be correct. Once a test mail is received, the configuration is correct and other notifications should work as well.
  • The vovnotifyd daemon uses operating system user names as the basis for email addresses. If this user name does not resolve in the email system, an email address map will need to be specified in the notification configuration.
Cannot see a specific tag anywhere in the interface As described in General Monitoring Configuration, tags that are not actively being monitored via sampling are hidden from the user interface by default. To ensure visibility for tags that fall under these conditions, use the setTagAccess configuration procedure as defined in Security.