Automate Cloud Scaling for Grid Engine

Create rules to automatically scale-up and scale-down deployment of nodes on the cloud. Use node configuration to customize a node associated with a Grid Engine cluster.

Define automation rules to filter resource requests from Grid Engine and route them to target instance types. You can provide rapid scaling parameters to scale-up node deployment on the cloud to meet a surge in demand for resources. After the jobs are finished, the nodes are undeployed based on a scale-down rule. Automations can also be added to a cluster after the cluster is deployed.
  1. Login to NavOps.
  2. Click Automations.
  3. Click Add Automation.
  4. Provide a name for the automation.
  5. Enter a description for the automation.
  6. Select a Grid Engine cluster.
  7. In the IF condition menu :
    • select graphql to filter job for dynamic scale-up automations..
    • select none to trigger actions without conditions. For example, if you want to set a rule to scale-up instances at 8 AM.
    • select node-filter to filter on the state of the node in the inventory. For example, used to scale-down instances once they are idle.
  8. For Jobs data driven scaling, in the If form field, enter a GraphQL statement to filter a set of PBS Pro jobs.
    Tip: Click (?) get more information about the supported schema. Here is an example to filter Queued Jobs Filter:
    orderBy: J_PRIORITY_DESC, filter: {states: [ 0 ], withSubJobs: true}
  9. The Trigger type is defined as calender.
  10. In the When section, select the values from the drop down menus to build the required cron expression displayed in the textbox form field below the menus. You can also enter valid chron expression directly in the textbox form field. For example, * * * * * will run the automation every minute and is the recommended default.
  11. In the Then select Job Data Driven Scaling and define the parameters:
    1. Select a Cost center.
    2. Select a Node configuration.
    3. Select the Node Class that will scale based on the defined rule for the accepted jobs.
      Note: Ensure that the NavOps server and PBS server IP addresses are configured in the cloud-init script of the selected node class.
    4. For Max. nodes per automation, maximum. number of nodes that can be deployed by this automation.
    5. For Max. nodes per automation run, enter the maximum number of nodes to be added at a time before reaching the target number of nodes.
    6. For Cool down, enter time to wait in seconds before adding the next set of nodes.
      Note: Typically, you may want to adjust the idle time for unburst. In certain situations, a lower restriction may be desirable to reduce expenses. The default value is -120s, which eliminates nodes that are idle for longer than two minutes. Here, values of -10s or lower are advised.
    7. Enter the number of tasks per nodes. This enables the scaling decision to understand the number of nodes to deploy based on the slot available.
  12. Click Save.
    The new automation is displayed in the automations table.
  13. Enable the Automation Engine and enable the automation.

Create Scale Down Automations

Nodes can be removed by filtering the inventory based on the state of the node. The advanced condition editor provides several templates to generat a query for scale down automations.
  1. Login to NavOps.
  2. Click Automation.
  3. Click Add Automation.
  4. Provide a name for the automation.
  5. Enter a description for the automation.
  6. Select a Grid Engine cluster.
  7. In the IF condition menu select node-filter.
  8. In the Query field, click () to open the advanced condition editor.
  9. Select a template to configure a condition. For example, Remove nodes with age-exec service idle.
    The configuration fields are populated based on the template. You can modify them as per your requirements. The generated query is displayed.
  10. Click Save.
  11. Trigger type is defined as calender.
  12. In the When section, select the values from the drop down menus to build the required cron expression displayed in the textbox form field below the menus. You can also enter valid chron expression directly in the textbox form field.
  13. In the Then menu, select Scale down (AGE).
  14. Click Save.
    The new automation is displayed in the automations table.
  15. Enable the Automation Engine and enable the automation.