Where once the purview of research labs and high-transaction applications, high performance computing (HPC) is now making its way into mainstream IT. Enterprises are finding out the value of analytics for keeping abreast of their customers and ahead of the competition. Many of these companies are now running their own HPC environments, supported by local parallel file system infrastructures. DDN’s WOS-Bridge can integrate high-performance computing with object storage and the cloud to help customers make commercial HPC cost effective and compliant.
In order to fully leverage their HPC investments, enterprises often must share their data across the company and even around the world. The commercial HPC use case needs a cloud-based storage infrastructure to foster this collaboration, and to facilitate the protection that these valuable data sets require. They also need to provide a way to archive the results and to store the ever-expanding primary data sets that are the raw materials for these HPC engines, since these analytics are often run repeatedly over time.
WOS-Bridge is the data migration engine that connects DDN’s GPFS and Lustre storage solutions with the WOS Cloud (and with each other) through a single global namespace. It automates and manages data movement between these traditional high performance storage platforms and the WOS Cloud through data replication and erasure coding technologies.
WOS-Bridge enables DDN’s WOS object storage to now act as an integrated local or distributed active archive tier behind their high performance storage environment. This “federated tiering” of data sets between local and cloud-based storage infrastructures can drive capacity cost savings and support active archiving, workflow collaboration and remote site disaster recovery protection.
By connecting to DDN’s GRIDScaler and EXAScaler parallel file system appliances WOS-Bridge allows users to move both DDN GPFS and Lustre data into the WOS object storage system that comprises a company’s active archive or private cloud. Users without access to the parallel file system can access the data in the WOS cloud through the WOS Access gateway, which provides POSIX-compliant NFS and CIFS access, or through other RESTful applications.
WOS Object Storage
Web Object Scaler (WOS) is DDN’s object storage system that’s behind many of the largest cloud infrastructures in existence today. In keeping with DDN’s high performance computing focus, WOS technology was designed to scale to extreme proportions with minimal performance impact. To this end, WOS maintains a map, in memory, of all instances of all objects within each node that defines the lowest latency path to that object. This ensures the best access performance regardless of where the requested data is located.
WOS’s “NoFS” architecture doesn’t use a file system as a means to organize data objects within the cluster of nodes, as many other object-based storage systems do. According to DDN, this process reduces overhead in the system and improves performance and scalability without compromising data reliability. By leveraging a combination of local and global erasure coding plus replication, WOS can provide data protection for geographically dispersed infrastructures without the high latency that many object storage systems incur.
High performance computing doesn’t exist in isolation; it’s part of a larger workflow that includes more commercial or mainstream applications and data handling tasks. Collaboration between departments or company locations using HPC-generated data sets is common, as are the requirements to store, protect and provision all the data used in this process, not just the results. Connecting HPC infrastructures to a High Performance Cloud with a solution like the WOS-Bridge, can help enterprises handle these data sets efficiently and more easily realize the benefits of high performance computing.
DataDirect Networks is not a client of Storage Switzerland