Automate Cloud Scaling for PBS Professional

Create rules to automatically scale-up and scale-down deployment of nodes on the cloud. Use node configuration to customize a node associated with a PBS cluster.

Define automation rules to filter resource requests from PBS Pro and route them to target instance types. You can provide rapid scaling parameters to scale-up node deployment on the cloud to meet a surge in demand for resources. After the jobs are finished, the nodes are undeployed based on a scale-down rule. Automations can also be added to a cluster after the cluster is deployed.
  1. Login to NavOps.
  2. Click Automation.
  3. Click Add Automation.
  4. Provide a name for the automation.
  5. Enter a description for the automation.
  6. Select a PBS cluster.
  7. In the IF condition menu :
    • select graphql to filter for dynamic scale-up automations..
    • select none to trigger actions without conditions. For example, if you want to set a rule to scale-up instances at 8 AM.
    • select node-filter to filter on the state of the node in the inventory. For example, used to scale-down instances once they are idle.
  8. For Simulation driven scaling, in the If form field, enter a GraphQL statement to filter a set of PBS Pro jobs.
    Tip: Click (?) get more information about the supported schema. Here is an example to filter Jobs Queued in "workq" Filter:
    orderBy: J_PRIORITY_DESC, filter: {queue: "workq", states: [ 0 ], withSubJobs: true}
  9. The Trigger type is defined as calender.
  10. In the When section, select the values from the drop down menus to build the required cron expression displayed in the textbox form field below the menus. You can also enter valid chron expression directly in the textbox form field. For example, * * * * * will run the automation every minute and is the recommended default.
  11. In the Then menu, select Simulation Driven Scaling and define the parameters:
    1. Select a Cost center.
    2. Select a Node configuration.
    3. Select the Node Class that will scale based on the defined rule for the accepted jobs.
      Note: Ensure that the NavOps server and PBS server IP addresses are configured in the cloud-init script of the selected node class.
    4. For Max. nodes per automation, enter the maximum number of nodes that can be deployed by this automation..
    5. For Max. nodes per automation run, enter the maximum number of nodes to be added at a time before reaching the target number of nodes.
    6. For Cool down, enter time to wait in seconds before adding the next set of nodes. This should be more than the time it takes for an instance to launch, connect to the WLM, and begin running jobs. For instance, a value of 90 seconds if something normally takes 70 seconds.
  12. Click Save.
    The new automation is displayed in the automations table.
  13. Enable the Automation Engine and enable the automation.

Create Scale Down Automations

Nodes can be removed by filtering the inventory based on the state of the node. The advanced condition editor provides several templates to generat a query for scale down automations.
  1. Login to NavOps.
  2. Click Automation.
  3. Click Add Automation.
  4. Provide a name for the automation.
  5. Enter a description for the automation.
  6. Select a PBS cluster.
  7. In the IF condition menu select node-filter.
  8. In the Query field, click () to open the advanced condition editor.
  9. Select a template to configure a condition. For example, Remove nodes with pbspro-mom service idle.
    The configuration fields are populated based on the template. You can modify them as per your requirements. The generated query is displayed.
    Note: Typically, you may want to adjust the idle time for unburst, which is identified in this table as agent-services.pbspro-mom.timestamp (using its meta-data label). In certain situations, a lower restriction may be desirable to reduce expenses. The default value is -120s, which eliminates nodes that are idle for longer than two minutes. Here, values of -10s or lower are advised.
  10. Click Save.
  11. Trigger type is defined as calender.
  12. In the When section, select the values from the drop down menus to build the required cron expression displayed in the textbox form field below the menus. You can also enter valid chron expression directly in the textbox form field.
  13. In the Then menu, select Scale down (PBS Pro).
  14. Click Save.
    The new automation is displayed in the automations table.
  15. Enable the Automation Engine and enable the automation.

Configure Snapshots of Automations for PBS Professional

The NavOps agent for PBS Professional supports a debug mode. When simulation-based scale-up is used, the snapshots used for automation runs can be retained for debugging purposes.

To enable snapshots for automations:

  1. Edit /opt/navops/etc/navops-actions-pbspro.yaml. 
    actions:
...
  - type: scale-up-pbspro
    ...
    config:
      pbs_sim_snapshot_retention: 5
    pbs_sim_snapshot_retention is an integer value which indicates how many snapshots will be saved for each automation. Older snapshots will be removed. If the value of this setting is 0 (or less), snapshots will not be saved.

    Snapshots are saved in $SNAPSHOT_WORK_DIR/<automation-uid>/ directories (one directory for each automation). For example, for an automation with UID of 7bad2a0d-6f65-48ab-b6fc-80a412e07478, the snapshots for that automation will be found in $SNAPSHOT_WORK_DIR/7bad2a0d-6f65-48ab-b6fc-80a412e07478/.

    If you enable snapshot retention for a period of time, and then disable it, the previously saved snapshots and directories will not be removed. Additionally, if you add/remove automations, the snapshot directories for removed automations will not be automatically removed. It is important to pay attention to available disk space when enabling this feature.

  2. Restart the agent using: 
    systemctl restart navops-agent
    or the PBS Professional actions service using:
    systemctl restart navops-actions-pbspro