High Performance Computing (HPC) environments have unique characteristics and some unique challenges that make them different from ‘traditional’ IT environments. HPC usually means big analytics projects that involve lots of data, high speed processing and frequently a compute infrastructure that’s been specifically designed for the project, often using open source software. HPC typically uses a ‘divide and conquer’ approach that leverages a parallel processing architecture where very large analytics problems are parsed into many smaller processing jobs and run on individual compute nodes, simultaneously.
High performance computing is very different from traditional IT processes, as is the HPC infrastructure itself, which is often physically separate from the ‘mainstream’ IT infrastructure. Its clustered compute nodes and supporting storage systems are designed to process the raw data it’s given and export the solution set as fast as possible, often using a distributed or parallel file system as opposed to NFS. At this point, these ‘temporary’ data storage pools are merged into the ‘persistent’ data storage that makes up the mainstream IT environment, for use in business intelligence (BI) applications, research, etc.
For corporate IT, creating a separate HPC environment is inefficient and dealing with open source computing systems can be unfamiliar. But given the growing importance of HPC and the HPC-like architectures that often come with projects such as Big Data, IT organizations are being forced to get familiar with HPC and learn how to manage its unique data storage requirements. Like it or not, HPC is making its way into mainstream IT and becoming a critical tool as companies strive to improve their competitiveness. This paper will examine the challenges of running an HPC project in an enterprise IT environment and how the storage infrastructure can evolve to help address those challenges.
This is an excerpt from the executive summary of Storage Switzerland’s latest white paper “The Evolution of HPC Storage Architectures in Commercial Environments”. To receive your exclusive copy, please fill out the form below.
Your message has been sent
HDS is a client of Storage Switzerland
Eric is an Analyst with Storage Switzerland and has over 25 years experience in high-technology industries. He’s held technical, management and marketing positions in the computer storage, instrumentation, digital imaging and test equipment fields. He has spent the past 15 years in the data storage field, with storage hardware manufacturers and as a national storage integrator, designing and implementing open systems storage solutions for companies in the Western United States. Eric earned degrees in electrical/computer engineering from the University of Colorado and marketing from California State University, Humboldt. He and his wife live in Colorado and have twins in college.
StorageSwiss.com - The Home of Storage Switzerland 