Automate Cloud Scaling for Slurm

Create rules to automatically scale-up and scale-down deployment of nodes on the cloud. Use node configuration to customize a node associated with a Slurm cluster.

Define automation rules to filter resource requests from Slurm and route them to target instance types. You can provide rapid scaling parameters to scale-up node deployment on the cloud to meet a surge in demand for resources. After the jobs are finished, the nodes are undeployed based on a scale-down rule. Automations can also be added to a cluster after the cluster is deployed.
  1. Login to NavOps.
  2. Click Automations.
  3. Click Add Automation.
  4. Provide a name for the automation.
  5. Enter a description for the automation.
  6. Select a Slurm cluster.
  7. In the IF condition menu :
    • select slurm job filter to filter job for dynamic scale-up automations.
    • select none to trigger actions without conditions. For example, if you want to set a rule to scale-up instances at 8 AM.
    • select node-filter to filter on the state of the node in the inventory. For example, used to scale-down instances once they are idle.
  8. Enter a query to filter a set of Slurm jobs.
    Tip: Here is an example:
    job_state=PENDING
    This filter finds Pending jobs in SLURM and passes them to the automation. There is a limit of 25 jobs. You can extend this by adding a page_size to the request. For example,
    job_state=PENDING&page_size=100
    This will find the top 100 Pending jobs in SLURM. If the page_size value exceeds a maximum limit of 200, the filter will not return jobs.
  9. The Trigger type is defined as calender.
  10. In the When section, select the values from the drop down menus to build the required cron expression displayed in the textbox form field below the menus. You can also enter valid chron expression directly in the textbox form field. For example, * * * * * will run the automation every minute and is the recommended default.
  11. In the Then select Jobs data driven scaling and define the parameters:
    1. Select a Cost center.
    2. Select a Node configuration.
    3. Select the Node Class that will scale based on the defined rule for the accepted jobs.
      Note: Ensure that the NavOps server and Slurm server IP addresses are configured in the cloud-init script of the selected node class.
    4. For Max. nodes per automation, maximum. number of nodes that can be deployed by this automation.
    5. For Max. nodes per automation run, enter the maximum number of nodes to be added at a time before reaching the target number of nodes.
    6. For Cool down, enter time to wait in seconds before adding the next set of nodes.
      Note: Typically, you may want to adjust the idle time for unburst. In certain situations, a lower restriction may be desirable to reduce expenses. The default value is -120s, which eliminates nodes that are idle for longer than two minutes. Here, values of -10s or lower are advised.
    7. Enter the Number of tasks per nodes. This enables the scaling decision to understand the number of nodes to deploy based on the slot available.
  12. Click Save.
    The new automation is displayed in the automations table.
  13. Enable the Automation Engine and enable the automation.

Create Scale Down Automations

Nodes can be removed by filtering the inventory based on the state of the node. The advanced condition editor provides several templates to generat a query for scale down automations.
  1. Login to NavOps.
  2. Click Automation.
  3. Click Add Automation.
  4. Provide a name for the automation.
  5. Enter a description for the automation.
  6. Select a Grid Engine cluster.
  7. In the IF condition menu select node-filter.
  8. In the Query field, click () to open the advanced condition editor.
  9. Select a template to configure a condition.
    The configuration fields are populated based on the template. You can modify them as per your requirements. The generated query is displayed.
  10. Click Save.
  11. Trigger type is defined as calender.
  12. In the When section, select the values from the drop down menus to build the required cron expression displayed in the textbox form field below the menus. You can also enter valid chron expression directly in the textbox form field.
  13. In the Then menu, select Scale down (Slurm).
  14. Enter the cool down time in seconds.
  15. Click Save.
    The new automation is displayed in the automations table.
  16. Enable the Automation Engine and enable the automation.