Cloud Scaling Overview

An introduction to cloud scaling, a typical scaling setup and an overview of steps to set up cloud scaling.

Cloud scaling is a configuration that is set up between an HPC cluster and a public cloud to deal with peaks in cluster demand. When resource capacity at the HPC reaches a certain point, the demand is directed to a public cloud so there is no interruption of services.

Cloud scaling requires that you already have an account with one of the supported cloud providers:
  • Microsoft Azure
  • Amazon Web Services (AWS)
  • Google Cloud Platform (GCP)
  • Oracle Cloud Infrastructure (OCI)

Cloud scaling is managed by integrating NavOps with the relevant workload manager and the NavOps Agent. The system links demand to resources by filtering incoming resource requests and routing them to benchmarked instance types on the cloud based on conditions defined in the automation engine. Refer to Automate Cloud Scaling for Accelerator, Automate Cloud Scaling for PBS Professional and Automate Cloud Scaling for Grid Engine Using JQ Filters for cloud scaling infomration relevant to the supported workload managers.

Steps for Cloud Scaling

There is some setup required at the cloud provider level before a cloud scaling can be achieved using NavOps. There is also NavOps configuration that must be done. General steps that are required for scaling:

  1. Create a cloud-init script that runs after a cloud node is deployed.
  2. Create necessary cloud provider components such as a virtual network, a subnet and a Virtual Machine (VM) image. To create the VM image, create a VM, customize it and create an image based on the VM.
  3. SSH into the cloud provider virtual machine and configure the VM for your site's environment such as installing NavOps, mounting file systems, connecting it to the authentication service, installing any applications, etc.
  4. Create an image of the virtual machine.
  5. Connect your cloud credential and then create an environment and node class with NavOps that provides the necessary cloud provider details required for cloud scaling.
  6. Add and deploy a cluster by installing NavOps CLI, NavOps agent, and add node configurations with startup scripts. Node configurations are not required for Accelerator.
  7. Create an automation.
Note: Steps vary depending upon the cloud provider used for cloud scaling.