Automate Cloud Scaling for Grid Engine Using JQ Filters

Create rules to automatically scale-up and scale-down deployment of nodes on the cloud. Use node configuration to customize a node associated with a Grid Engine cluster. Add suitable host groups and resources. For example, cloud_job = true, h_vmem, license, ncpus, and slots.

Enable JQ Filters for Automations

Administrator privileges are required to enable the JQ filters (Scale up job filter).
  1. As Administrator, edit the navops-agent.yaml file located at /opt/navops/etc.
  2. Uncomment the following lines. 
    - name: conditions-wlm
type: external
configuration:
status_url: http://127.0.0.1:8083/v1/status
managed: true
unit_name: navops-conditions-wlm
  3. Restart the agent: systemctl restart navops-agent.
    A new service called conditions-wlm is enabled and the Scale up job filter condition type for JQ filters is available for Automations.

Add Automations

Define automation rules to filter resource requests from Grid Engine and route them to target instance types. You can specify job query filters for automations using JQ and GraphQL.You can provide rapid scaling parameters to scale-up node deployment on the cloud to meet a surge in demand for resources. After the jobs are finished, the nodes are undeployed based on a scale-down rule. Automations can also be added to a cluster after the cluster is deployed.
  1. Login to NavOps.
  2. Click Automations.
  3. Click Add Automation.
  4. Provide a name for the automation.
  5. Enter a description for the automation.
  6. Select a Grid Engine cluster.
  7. In the IF condition menu :
    • select Scale up job filter to filter job for dynamic scale-up automations..
    • select none to trigger actions without conditions. For example, if you want to set a rule to scale-up instances at 8 AM.
    • select node-filter to filter on the state of the node in the inventory. For example, used to scale-down instances once they are idle.
  8. For Jobs data driven scaling, in the If form field, enter a JQ statement to filter a set of Grid Engine jobs.
  9. Filters you can define: 
    <field_name>__contains=<substring-value>
<field_name>__in=<exact-string-value1>,<exact-string-value2>, ...
<field_name>__gt=<numeric-value>
<field_name>__gte=<numeric-value>
<field_name>__lt=<numeric-value>
<field_name>__lte=<numeric-value>
<field_name>__startswith=<substring-value>
<field_name>__endswith=<substring-value>
    Note: Rules can be chained with & (but not with | (or) ).
  10. Sorting options available are: 
    sort=<field_name>
sort=<field_name>__asc
sort=<field_name>__desc
Other:
page_size=<num of record>
  11. Here are some examples of filters: 
    Queued Jobs:
state=qw

Job Class optistruct requested (using contains as job has optistruct.default):
jclass_name__contains=optistruct

Filter for no job class requested (empty):
jclass_name=

Filter for hard request for complex ncpus >=1 and <=4:
rd_list.job_hard_resources.ncpus__gte=1&rd_list.job_hard_resources.ncpus__lte=4

Filter for soft request for complex ncpus >2 and <=4:
rd_list.job_soft_resources.ncpus__gt=2&rd_list.job_soft_resources.ncpus__lte=8
Note: We should always have a >0 filter as empty values are <x where x>0, e.g.
rd_list.job_soft_resources.ncpus__gt=0&rd_list.job_soft_resources.ncpus__lte=8

Filter for Parallel Environment (PE) and slots requested in PE:
requested_pe.name=pe1&requested_pe.pe_min__lgt=4&requested_pe.pe_max__lte=8

For MPI PE Type Jobs:
state=qw&requested_pe.name=pe*
state=qw&requested_pe.name__contains=pe
state=qw&requested_pe.name__in=pe1,pe2,pe3

Jobs that don't request PE:
requested_pe.name=&state=qw

Filter for Project:
JB_project=engineering

Filter for Requested Queue:
user_hard_requested_queues=all.q
user_hard_requested_queues__contains=all
user_soft_requested_queues__contains=cloud

Filter for jobs older than 3 minutes (units is seconds so >180):
JB_job_age__gt=180

Complex Examples Derived From Above:
jclass_name__contains=optistruct&rd_list.job_hard_resources.ncpus__gt=2
&rd_list.job_hard_resources.ncpus__lte=4
jclass_name=&rd_list.job_hard_resources.ncpus__gt=2
&rd_list.job_hard_resources.ncpus__lte=4
jclass_name__in=nastran.default,optistruct.default
jclass_name__in=nastran.default,optistruct.default
&rd_list.job_hard_resources.ncpus__gt=2
&rd_list.job_hard_resources.ncpus__lte=4
state=qw&JB_job_age__gt=180&jclass_name__contains=optistruct
&rd_list.job_hard_resources.ncpus__gte=1
&rd_list.job_hard_resources.ncpus__lte=4
state=qw&requested_pe.name=pe1&requested_pe.requested_pe=8
state=qw&requested_pe.name__in=pe1,pe*

Example sorting by JobID (oldest jobs at top) and Top 20 oldest jobs:
sort=JB_job_number__asc&jclass_name__contains=nastran&page_size=20
    Note: Additionally, depending on the format of the job data, it may be also be necessary to:
    • Modify the jq-based data transformation being performed on the job data in /opt/navops/bin/get-job-data.sh.
    • Modify the field_mapping in /opt/navops/etc/navops-conditions-wlm.yaml.
  12. The Trigger type is defined as calender.
  13. In the When section, select the values from the drop down menus to build the required cron expression displayed in the textbox form field below the menus. You can also enter valid chron expression directly in the textbox form field. For example, * * * * * will run the automation every minute and is the recommended default.
  14. In the Then select Jobs Data Driven Scaling and define the parameters:
    1. Select a Cost center.
    2. Select a Node configuration.
    3. Select the Node Class that will scale based on the defined rule for the accepted jobs.
      Note: Ensure that the NavOps server and Grid Engine server IP addresses are configured in the cloud-init script of the selected node class.
    4. For Max. nodes per automation, maximum. number of nodes that can be deployed by this automation.
    5. For Max. nodes per automation run, enter the maximum number of nodes to be added at a time before reaching the target number of nodes.
    6. For Cool down, enter time to wait in seconds before adding the next set of nodes.
      Note: Typically, you may want to adjust the idle time for unburst. In certain situations, a lower restriction may be desirable to reduce expenses. The default value is -120s, which eliminates nodes that are idle for longer than two minutes. Here, values of -10s or lower are advised.
    7. Enter the number of tasks per nodes. This enables the scaling decision to understand the number of nodes to deploy based on the slot available.
  15. Click Save.
    The new automation is displayed in the automations table.
  16. Enable the Automation Engine and enable the automation.

Create Scale Down Automations

Nodes can be removed by filtering the inventory based on the state of the node. The advanced condition editor provides several templates to generat a query for scale down automations.
  1. Login to NavOps.
  2. Click Automation.
  3. Click Add Automation.
  4. Provide a name for the automation.
  5. Enter a description for the automation.
  6. Select a Grid Engine cluster.
  7. In the IF condition menu select node-filter.
  8. In the Query field, click () to open the advanced condition editor.
  9. Select a template to configure a condition. For example, Remove nodes with age-exec service idle.
    The configuration fields are populated based on the template. You can modify them as per your requirements. The generated query is displayed.
  10. Click Save.
  11. Trigger type is defined as calender.
  12. In the When section, select the values from the drop down menus to build the required cron expression displayed in the textbox form field below the menus. You can also enter valid chron expression directly in the textbox form field.
  13. In the Then menu, select Scale down (AGE).
  14. Click Save.
    The new automation is displayed in the automations table.
  15. Enable the Automation Engine and enable the automation.