HPC
BeeGFS Storage Pools: Exploring the Benefits of Multi-Tiered HPC Storage
August 27, 2019
3 min read
Optimize HPC Storage Using BeeGFS Storage Pools
Storage pools were introduced with BeeGFS v7.0 and with it, allow the cluster administrator to group storage targets and mirror buddy groups together in specific classes. For example, there can be one pool consisting of fast, but small solid state drives, and another pool for bulk storage, using big but slower spinning disks. Pools can have descriptive names, making it easy to remember which pool to use without looking up the storage targets in the pool. The SSD pool could be named "fast" and the other "bulk".
While all-flash systems usually are still too expensive for systems that require large capacity, a certain amount of flash drives are typically affordable in addition to the spinning disks for high capacity. The goal of a high-performance storage system should then be to take optimal advantage of the flash drives to provide optimum access speed for the projects on which the users are currently working.
| FLASH-BASED | HDD-BASED |
| BeeGFS-storage targets can be combined into a "Fast Pool" and tied to a directory used for active/ higher-performance I/O work. | BeeGFS-storage targets can be combined (or left in the default storage pool) and used in either a defined directory or as the default for the rest of the BeeGFS storage pool for archiving or work that doesn't need to be done on the "Fast Pool". |
Taking Full Advantage of Flash Storage
To enable users to get the full, all-flash performance for the projects on which they are currently working, the BeeGFS storage pools feature makes the flash drives explicitly available to the users. This way, users can request from BeeGFS to move the current project to the flash drives and thus all access to the project files will be served directly and exclusively from the flash drives without any access to the spinning disks until the user decides to move the project back to the spinning disks. The placement of the data is fully transparent to applications. Data stays inside the same directory when it is moved to a different pool and files can be accessed directly without any implicit movement, no matter which pool the data is currently assigned to. To prevent users from putting all their data on the flash pool, different quota levels exist for the different pools, based on which a sysadmin could also implement a time-limited reservation mechanism for the flash pool.
Tying It All Together
However, while the concept of storage pools were originally developed to take optimum advantage of flash drives in a system with high capacity based on spinning disks, the concept is not limited to this particular use-case. The system admin can group arbitrary storage targets in such pools, no matter which storage media they use and there can also be more than just two pools.
For a Hands on Tutorial on How to Create & Implement Storage Pools in BeeGFS V7, Click Below!
HPC
BeeGFS Storage Pools: Exploring the Benefits of Multi-Tiered HPC Storage
August 27, 20193 min read
Optimize HPC Storage Using BeeGFS Storage Pools
Storage pools were introduced with BeeGFS v7.0 and with it, allow the cluster administrator to group storage targets and mirror buddy groups together in specific classes. For example, there can be one pool consisting of fast, but small solid state drives, and another pool for bulk storage, using big but slower spinning disks. Pools can have descriptive names, making it easy to remember which pool to use without looking up the storage targets in the pool. The SSD pool could be named "fast" and the other "bulk".
While all-flash systems usually are still too expensive for systems that require large capacity, a certain amount of flash drives are typically affordable in addition to the spinning disks for high capacity. The goal of a high-performance storage system should then be to take optimal advantage of the flash drives to provide optimum access speed for the projects on which the users are currently working.
| FLASH-BASED | HDD-BASED |
| BeeGFS-storage targets can be combined into a "Fast Pool" and tied to a directory used for active/ higher-performance I/O work. | BeeGFS-storage targets can be combined (or left in the default storage pool) and used in either a defined directory or as the default for the rest of the BeeGFS storage pool for archiving or work that doesn't need to be done on the "Fast Pool". |
Taking Full Advantage of Flash Storage
To enable users to get the full, all-flash performance for the projects on which they are currently working, the BeeGFS storage pools feature makes the flash drives explicitly available to the users. This way, users can request from BeeGFS to move the current project to the flash drives and thus all access to the project files will be served directly and exclusively from the flash drives without any access to the spinning disks until the user decides to move the project back to the spinning disks. The placement of the data is fully transparent to applications. Data stays inside the same directory when it is moved to a different pool and files can be accessed directly without any implicit movement, no matter which pool the data is currently assigned to. To prevent users from putting all their data on the flash pool, different quota levels exist for the different pools, based on which a sysadmin could also implement a time-limited reservation mechanism for the flash pool.
Tying It All Together
However, while the concept of storage pools were originally developed to take optimum advantage of flash drives in a system with high capacity based on spinning disks, the concept is not limited to this particular use-case. The system admin can group arbitrary storage targets in such pools, no matter which storage media they use and there can also be more than just two pools.
For a Hands on Tutorial on How to Create & Implement Storage Pools in BeeGFS V7, Click Below!
