High-Performance Computing Architecture for Cloud Deployment

High-performance computing (HPC) allows you to process data and perform complex calculations quickly. This article discusses what to consider when building your high-performance computing architecture for cloud deployment.

Traditionally, the capacity of your on-premises infrastructure limited HPC systems. Today, the cloud allows you to extend local capacity with resources in the cloud.

HPC workloads are well-suited for pay-as-you-go cloud infrastructure because they fluctuate and burst. Fine-tuned cloud resources and cloud-native architecture can speed up the turnaround of HPC workloads.

You can't utilize on-premises HPC solutions sized for peak capacity until they reach peak capacity. Otherwise, they will be underutilized and cost your organization money. A cloud-based HPC infrastructure can mitigate the risk of missing significant opportunities when you’re working on time-sensitive jobs. High-performance computing architecture can also eliminate the cost of unused infrastructure.

Design Principles for High-Performance Computing Architecture

Below is a set of general design principles for HPC architecture in the cloud:

Dynamic Architecture

Your architecture should be dynamic in that it can grow and shrink to match your demands for HPC capacity over time. When designing your architecture, consider the natural cycles of your HPC activity. 

In the design phase, for example, you might see an increase in demand followed by a decrease in demand as the project moves to another stage. A lot of HPC projects are bursty and well-suited to cloud paradigms. You don’t have to choose between oversubscribed or idle systems when you have elasticity and pay-as-you-go provided by cloud.

Data Visibility

You need a clear picture of your data—data origin, size, velocity, and updates—before you design your HPC architecture. The goal is to optimize performance and cost holistically. Consider using your cloud vendor’s data services, such as data visualization, which enables you to extract the most value from your data.

Collaboration

Many HPC projects are collaborative, sometimes spanning multiple countries. Besides collaboration on a specific project, methods and results are often shared with the larger HPC and scientific communities. Choosing in advance which tools, scripts, and data to share and with whom is essential. During the design process, you should consider project delivery methods.

Cloud-Native Design

Replicating your on-premises environment is usually unnecessary and not optimal when you migrate workloads to the cloud. With cloud services, HPC workloads can run in new ways using cloud-native design patterns and solutions. Automating HPC cluster deployment allows you to quickly end one compute cluster and launch a new one.

Workload Testing

Testing your workload in the cloud is the best way to know how it will perform. There are a lot of complex HPC applications, and you can’t run a simple test on their memory, CPU, and network patterns. 

In addition, your infrastructure requirements depend on the algorithms you use as well as the size and complexity of your models. As a result, generic benchmarks can’t reliably predict HPC performance. In the cloud, you only pay for what you use so that you can create a realistic proof of concept. One advantage of a cloud-based platform is that you can run a full-scale test before you migrate.

Cost vs. Time

With high-performance computing architecture, you can analyze performance based on time and cost. You should optimize workloads that aren’t time-sensitive for cost. The least expensive way to run non-time-critical workloads is with spot or preemptible instances. Conversely, for time-critical workloads, performance should take precedence over cost optimization. In this case, you should pick the instance type, procurement model, and cluster size with the quickest execution time. When comparing cloud platforms, you should consider non-computing factors such as provisioning, staging data, or time spent in queues.