High-Performance Computing (HPC) is no longer a workload for academia, now organizations of all types and sizes have HPC like workloads that require scalable, sequential access to vast amounts of unstructured data. These initiatives need rapid access and need to respond to the requesting application seamlessly. The storage architecture requires a unique combination of scale, high performance, mixed workload support and cost-effective long-term retention. Most IT teams count on the enterprise storage systems they already have to meet these demands, and most are disappointed with the results.
Traditional HPC Storage Pushes IT to Enterprise Storage
The reason that IT tries to stretch their enterprise solutions to cover the HPC use case is that most HPC storage solutions appear to be too much of a science project. Enterprise IT doesn’t have the time to source software and hardware from multiple sources. They perceive their enterprise solution to be the turnkey solution to the problem.
Why Traditional Storage Falls Short
Commercial HPC typically stores a mixture of very large files as well as very small files. In the typical HPC project, there are millions if not billions of small files that require analysis. Typically, there are dozens, if not hundreds, of compute nodes accessing HPC data to perform this analysis. Managing the IO requests from hundreds of compute nodes as well as managing the access to the mixed file sizes is enough to cripple the typical storage controller or filer head. Even scale-out systems get bottlenecked in managing all the accesses since most systems have to route IO through a couple of control nodes.
Enterprises need an HPC storage system designed specifically for the Commercial HPC use case. One that is turnkey for rapid deployment and time to value. It also needs to deliver the performance that Commercial HPC requires by providing parallel access to data from any compute node.
To learn more about designing storage architectures for commercial HPC, join us for our on demand webinar, “Is Your Storage Ready for HPC? – Three Steps to Take“. Register for the webinar and get a copy of our white paper “Designing Storage for Commercial HPC” which is available exclusively to webinar registrants.