High-Performance Computing (HPC) is no longer a workload for academia, now organizations of all types and sizes have HPC like workloads that require scalable, sequential access to vast amounts of unstructured data. These initiatives need rapid access and need to respond to the requesting application seamlessly. The storage architecture requires a unique combination of scale, high performance, mixed workload support and cost-effective long-term retention. Most IT teams count on the enterprise storage systems they already have to meet these demands, and most are disappointed with the results.
Traditional HPC Storage Pushes IT to Enterprise Storage
The reason that IT tries to stretch their enterprise solutions to cover the HPC use case is that most HPC storage solutions appear to be too much of a science project. Enterprise IT doesn’t have the time to source software and hardware from multiple sources. They perceive their enterprise solution to be the turnkey solution to the problem.
Why Traditional Storage Falls Short
Commercial HPC typically stores a mixture of very large files as well as very small files. In the typical HPC project, there are millions if not billions of small files that require analysis. Typically, there are dozens, if not hundreds, of compute nodes accessing HPC data to perform this analysis. Managing the IO requests from hundreds of compute nodes as well as managing the access to the mixed file sizes is enough to cripple the typical storage controller or filer head. Even scale-out systems get bottlenecked in managing all the accesses since most systems have to route IO through a couple of control nodes.
StorageSwiss Take
Enterprises need an HPC storage system designed specifically for the Commercial HPC use case. One that is turnkey for rapid deployment and time to value. It also needs to deliver the performance that Commercial HPC requires by providing parallel access to data from any compute node.
To learn more about designing storage architectures for commercial HPC, join us for our on demand webinar, “Is Your Storage Ready for HPC? – Three Steps to Take“. Register for the webinar and get a copy of our white paper “Designing Storage for Commercial HPC” which is available exclusively to webinar registrants.
Watch On Demand
George Crump is the Chief Marketing Officer at VergeIO, the leader in Ultraconverged Infrastructure. Prior to VergeIO he was Chief Product Strategist at StorONE. Before assuming roles with innovative technology vendors, George spent almost 14 years as the founder and lead analyst at Storage Switzerland. In his spare time, he continues to write blogs on Storage Switzerland to educate IT professionals on all aspects of data center storage. He is the primary contributor to Storage Switzerland and is a heavily sought-after public speaker. With over 30 years of experience designing storage solutions for data centers across the US, he has seen the birth of such technologies as RAID, NAS, SAN, Virtualization, Cloud, and Enterprise Flash. Before founding Storage Switzerland, he was CTO at one of the nation's largest storage integrators, where he was in charge of technology testing, integration, and product selection.
StorageSwiss.com - The Home of Storage Switzerland 