How vovserver Failover Works

If the vovserver crashes, after a period of time, the vovtasker process on each machine notices that it has had no contact from the server, and it initiates a server machine election.

In this election, each vovtasker votes for itself (precisely, the host that this particular tasker runs on) as a server candidate. The election is conducted by running the script vovservsel.tcl.

After the time interval during which the vovtaskers vote expires, (default 60 seconds) the host that appears earliest on the list will be selected to start a new vovserver.

In the following example, the servercandidates.tcl, file contains three hosts: 
set ServerCandidates {
    host1
    host2
    host3
}
When the server crashes, if there are vovtaskers running on host1, host2 and host3, then these hosts will be voted as server candidates. Then host2 will be the best candidate and a new vovserver will be started on host2. This server will start in Crash Recovery Mode.
Note: For failover recovery to be successful, an active vovtasker process must be running on at least one of the hosts named in the ServerCandidates list. Usually, these vovtaskers have been defined with the -failover option so they can not accept any jobs, and are members of the failover taskergroup.

The failover vovserver will read the most-recently-saved PR file from the .swd/trace.db directory, and then read in the transactions from the 'cr*' (crash recovery) files to recover as much of the pre-crash state as possible.

The vovserver writes a new serverinfo.tcl file in the .swd that vovtaskers read to determine the port and host. When it starts, the failover vovserver appends the new host and port information to the $NC_CONFIG_DIR/<queue-name.tcl> as well as to the setup.tcl in the server configuration directory. The vovserver then runs the scripts in the Autostart Directory. This should include the failover.csh script, which resets the failover directory so that failover can repeat. This script removes the registry entry, and removes the server_election directory and creates a new empty one. At the end, it calls vovproject reread to force the failover vovserver to create an updated registry entry.

The failover vovserver remains in crash recovery mode for an interval, usually one minute, waiting for any vovtaskers that have running jobs to reconnect:
  • For Accelerator, Accelerator Plus, Monitor and Allocator, vovtaskers wait up to 4 days for a new server to start.
  • For FlowTracer, vovtaskers wait up to 3 minutes for a new server to start.

After reconnecting to vovserver, vovtaskers automatically exit after all of their running jobs are completed. After the vovserver transitions from crash recovery mode to normal mode, it will try to restart any configured vovtaskers that are not yet running.

Any of the following conditions will prevent successful failover server restart:
  • The filesystem holding the .swd directory is unavailable.
  • The file servercandidates.tcl does not exist.
  • The ServerCandidates list is empty.
  • There is no vovtasker running on any host in the ServerCandidates list when the server crashes.
  • The autostart/failover.csh script file is not in place.

In this case, the failover server will not be automatically started; the server will have to be manually started.

Tips for Configuring Failover

Following are tips for failover configuration:
  • Make the first failover host the regular one. This way, if the vovserver dumps core or is killed by mistake, it will restart on the regular host.
  • Configure special vovtaskers only for failover by passing the -failover option to vtk_tasker_define.
  • Test that failover works before depending on it.

Migrating vovserver to a New Host

The failover mechanism provides the underpinnings of a convenient user CLI command that can be used to migrate vovserver to a new host:
ncmgr rehost -host  NEWHOST
The specified NEWHOST must be one of the hosts eligible for failover of vovserver.