Optimize HPC Storage Using BeeGFS Storage Pools
Storage pools were introduced with BeeGFS v7.0 and with it, allow the cluster administrator to group storage targets and mirror buddy groups together in specific classes. For example, there can be one pool consisting of fast, but small solid state drives, and another pool for bulk storage, using big but slower spinning disks. Pools can have descriptive names, making it easy to remember which pool to use without looking up the storage targets in the pool. The SSD pool could be named “fast” and the other “bulk”.
While all-flash systems usually are still too expensive for systems that require large capacity, a certain amount of flash drives are typically affordable in addition to the spinning disks for high capacity. The goal of a high-performance storage system should then be to take optimal advantage of the flash drives to provide optimum access speed for the projects on which the users are currently working.
Taking Full Advantage of Flash Storage
To enable users to get the full, all-flash performance for the projects on which they are currently working, the BeeGFS storage pools feature makes the flash drives explicitly available to the users. This way, users can request from BeeGFS to move the current project to the flash drives and thus all access to the project files will be served directly and exclusively from the flash drives without any access to the spinning disks until the user decides to move the project back to the spinning disks. The placement of the data is fully transparent to applications. Data stays inside the same directory when it is moved to a different pool and files can be accessed directly without any implicit movement, no matter which pool the data is currently assigned to. To prevent users from putting all their data on the flash pool, different quota levels exist for the different pools, based on which a sysadmin could also implement a time-limited reservation mechanism for the flash pool.
Tying It All Together
However, while the concept of storage pools were originally developed to take optimum advantage of flash drives in a system with high capacity based on spinning disks, the concept is not limited to this particular use-case. The system admin can group arbitrary storage targets in such pools, no matter which storage media they use and there can also be more than just two pools.