Preemption Examples

Preemption by Priority

In the following example, all high priority jobs requesting License:abc can preempt all low priority jobs. It is implied that the preempted low priority job must provide the resource License:abc, otherwise it will not be preempted. 
# An implementation of the old "preemption by priority" policy.
VovPreemptRule -rulename "SimplePriority" \ 
      -ruletype    GENERIC  -preempting  "priority>=8"    -waitingfor License:abc \ 
      -preemptable "priority<4 "
The next example generalizes the preemption by priority, within a specific job class. In this case, the preempting job needs to be in the jobclass called hsim, and the preemptable jobs are selected only if the have the same jobclass as the preempting job: 
# A more generic preemption by priority for jobs in the hsim jobclass.
VovPreemptRule -rulename "Priority_hsim" \ 
      -ruletype    GENERIC  -preempting  "jobclass==hsim priority>=8" \ 
      -preemptable "jobclass==@JOBCLASS@ priority<@PRIORITY@"

Preemption Across Jobclasses

In this example, the jobclass urgent can preempt jobs in the jobclass regression but only for jobs that belong to the same project, as expressed by jobproj==@JOBPROJ@. Also, a relatively long kill age of 10 minutes is allowed; the preemptable jobs that are younger than 10 minutes are withdrawn and resubmitted, while jobs that are older are suspended and later resumed.
# A preemption between jobclasses, but within the same project
VovPreemptRule -rulename "Urgent_vs_regression" \ 
       -preempting  "jobclass==urgent"  -preemptable "jobclass==regression jobproj==@JOBPROJ@"  -killage 10m

Reserve Resources for Token-based Jobs

In the following example, a weak preemption type called RESERVE_RESOURCES is used to help dispatch jobs that require multiple tokens in the presence of other jobs that compete for the same tokens. In this case, if there is a job in the class ultrasim with priority greater than 4, the system reserves 6 tokens of whatever the job requires for 3 minutes. 
VovPreemptRule -rulename "UltrasimWeak" \ 
    -preempting "jobclass==ultrasim priority>4" \ 
    -ruletype RESERVE_RESOURCES -reservetime 3m -reservenum 6 \ 
    -pool Ultrasim -order 10
Note: There is no mention of the resource required by the jobs in the preempting jobclass, because the resources are computed automatically.
Such a rule (no mention of required resources) may be combined with a stronger rule. For example, such a rule can actively preempt other jobs by the same user, provided that the ultrasim job has been waiting for at least 4 minutes.
VovPreemptRule -rulename "UltrasimStronger" \ 
    -preempting "jobclass==ultrasim priority>4" -bucketage 4m \ 
    -preemptable "jobclass==spectre user==@USER@ priority<@PRIORITY@" \ 
    -pool Ultrasim -order 20

MultiQueue Preemption

MultiQueue is the older name for the system now called Allocator™. The preemption system still refers to these rules with the old name.

In the following example, assume that the resources WAN:abc and WAN:hsim are managed by Allocator. You want the preemption daemon to preempt a job that uses those resources when Allocator requires a "substantial reduction" in the number of available resources for this site. In the example, a different rule is created for each resource and each rule is assigned to a different pool. The substantial reduction is defined a 5% change in the allocated resources. 
foreach mqRes { 
    WAN:abc 
    WAN:hsim 
} {
    VovPreemptRule -rulename "MultiQueueLicense_$mqRes" \  
        -multiqueueres $mqRes  -mqthresh 0.05  -pool POOL$mqRes
}

Use the -resumeres Option

In many cases, the preemption occurs to make a saturated license available to an important job. Upon suspension of the preempted job, the license becomes available to the preempting job, while the resumer job, which also wants the same license, waits because it typically has lower priority than the preempting job.

In some cases, however, the license that is being preempted is not saturated, and preemption occurs on account of the lack of slots (also known as hardware preemption). In such case, the resumer job could be executed immediately on any available slot, which may be undesirable. The resources specified by the -resumeres option can be added to the resumer job as a way to better control its execution.
Note: When preemption occurs because of hardware, the -resumeres option is highly recommended.
Note: See @KEYWORD* Expansion for details on using @KEYWORD@ expansion with -resumeres.

For example, assume that jobs in the class C can only execute on taskers that offer the resource R. A low-priority job in the class C is preempted. The corresponding resumer job requires only the resources grabbed by the preempted job. Since the resumer job can execute on any host and the resources grabbed by the suspended job are not saturated, the resumer job fires immediately, leading to a premature resumption of the suspended job and to the possible overloading of the host on which the resumed job is running. However, if the preemption rule is specified with the option -resumeres HOST=@HOST@, then the resumer job is forced to execute only on host called @HOST@, which is mapped to the name of the host on which the preempted job was originally running.

All fields in the preempted job can be used in the -resumeres argument. The most useful are @TASKERNAME@ (or @NAME@ for short), @TASKERHOST@ (or @HOST@ for short) and @HWRAM@, @HWPERCENT@, @HWSLOTS@, @HWCPUS@, and @HWSWAP@, which are computed from the SOLUTION property of the preempted job. The open slot goes to the preempting job because of priority, and the resumer job waits for at least one slot to open up on the host of the suspended job. 
#
# The resumer job must execute on the same host as the
# preempted job.
#
VovPreemptRule -rulename XX  -ruletype    GENERIC  -preempting "jobclass==abc priority>=8" -waitingfor "HW"  -preemptable 
"jobclass==abc priority<4"  -resumeres "TASKERNAME=@TASKERNAME@ HOST=@HOST@ RAM/@HWRAM@ SLOTS/@HWSLOTS@"

#
# The resumer job must execute on the same tasker as the
# preempted job.
VovPreemptRule -rulename YY  -ruletype    GENERIC  -preempting "jobclass==abc priority>=8" -waitingfor "HW"  -preemptable 
"jobclass==abc priority<4"  -resumeres "TASKERNAME=@TASKERNAME@"

Preempt Ultrasim

Ultrasim uses Virtuoso_Multi_mode_Simulation, typically 4 or 6 tokens. To preempt Ultrasim, you can use this example: 
VovPreemptMethod License:Virtuoso_Multi_mode_Simulation SIGTSTP

VovPreemptRule -rulename ultrasim \ 
      -ruletype GENERIC \
      -killage    10 \ 
      -waitingfor "License:Virtuoso_Multi_mode_Simulation" \ 
      -preempting "priority>8" \ 
      -preemptable "priority<@PRIORITY@" \
      -method AUTOMATIC

Use Preemption to Reserve Taskers

In the following example, if there are scheduled jobs from the FairShare group "/class/ABCD", then we reserve 3 taskers called linux02, linux04, and linux06 for 1 minute. If, in that 1 minute, the tasker becomes available, the preempting job will have exclusive access to that tasker. Since the rule is recomputed every few seconds, it is sufficient to have short reserve times, such as one minute or even less. 

VovPreemptRule -rulename reserveHardware \ 
      -ruletype    RESERVE_TASKERS \
      -preempting "GROUP~/class/ABCD" \ 
      -reservetime 1m \
      -reservefor  "GROUP @FSGROUP@" \
      -reservetasker "linux02 linux04 linux06" 
      -reservenum 2