Even though the workloads are only a few years old, the typical file system used in AI and High Velocity Analytics was created decades ago. These file systems, while parallel in nature, were optimized for large file, high bandwidth environments, not the billions of small files environment characteristic of AI and High Velocity Analytics. The small file characteristics of these modern workloads require rethinking metadata management and even data protection strategies.
Legacy parallel file systems are also not typically optimized for modern advances in storage like NVMe SSD drives, NVMe over Fabric networks and multi-threaded servers which the file system uses as nodes. Most legacy file systems count on stock Linux drivers to provide support for such features. Modern file systems should create specific drivers and optimizations for these major moves forward in technology.
In Storage Switzerland’s latest eBook “AI and High-Velocity Analytics Workloads Need a New File System” we discuss the requirements of file systems that support these workloads. To get your copy register for our 15 minute webinar “15 Minute Webinar – Finding the Right File System for AI and ML Workloads” and the eBook will be available as an attachment as soon as it starts playing.