High Performance Computing (HPC) creates interesting challenges for the storage infrastructure. First, storage systems supporting HPC have to meet increasingly higher performance demands. High performance HPC storage enables more simulations and deeper analytics across a wider set of data. Leveraging a scale-out all-flash array can meet many of these performance requirements. The greater obstacle, however, is cost effectively dealing with the capacity challenges HPC places on the storage infrastructure.
Can Flash Do it All?
Some flash vendors claim a single flash system can meet both HPC performance AND capacity demands. Most scale-out flash systems should be able to scale to meet the performance demands of the typical HPC environment and, at least initially, should be able to scale to meet the capacity demands. Remember scale-out storage systems count on parallel workloads in order to deliver on their performance claims, something that the HPC environment tends to provide. The result is, technically, an all-flash, scale-out storage system could deliver both the capacity and performance HPC needs. The bigger question is can the organization afford to invest in all-flash for all of its HPC data, especially long term?
Justifying Hard Disks for HPC
For many HPC workloads there is time to move a data set from cost-effective, high capacity storage to high performance storage. If that data movement is simple and quick enough then the organization can see a significant cost savings by moving data not currently under modeling or analyzation to a scale-out, object based storage tier. Some object storage systems can add power savings capabilities, spinning down hard disk drives and even powering down nodes within the storage cluster.
An object based capacity tier also allows the primary HPC storage tier to be more flexible. Since it will deal with a much smaller, active data set only, the primary tier could use high performance scale-out all-flash NAS or block storage that effectively acts as a cache to the back-end object storage.
Creating a Bridge
The key is to create a bridge between an high performance HPC storage tier and its capacity tier. This means the organization either needs to manage the movement between tiers manually, invest in software that manages data movement between them or look for a storage system that can automatically move data between the two tiers. The effectiveness of this bridge is critical to the HPC seeing maximum benefit out of the capacity tier and enables them to keep the investment in the premium flash tier to a minimum.
Dive Deep on HPC Storage Infrastructure Design
Watch on our on demand webinar with Storage Switzerland and Caringo for a HPC-focused webinar. We dive deep on HPC storage infrastructure design, discuss the pros and cons to the various option in HPC storage infrastructure and provide key steps in developing a strategy that makes sense, not only for HPC but the rest of the enterprise as well.