linux administration: disk configuration for NVMe + HDD but better than BCache
Budget: $250 – $750 USD
The job is to find best technology for hybrid disk: 4TB NVMe (fast) + 2x 16TB HDD (slow). Challenge: maximum performance.
The combined disk array has to be as large as possible (min. 30TB) and as fast as possible both for READ IOPS and write throughput. This is for high-performance testing setup and not for critical production systems, so possible are not-to-frequent crashes or even some occasional data loss is ACCEPTABLE, if brings performance.
File level solutions like standard configuration of `BCache`does not handle out needs well. Because the usage on this array is multiple large database files (append-only and indexes R/W) or very large virtualization files in which VMs are run. So it is database-like intensive and random load. Secondary type of load is heavy duty software compilation and linking.
The typical load is 6-10x more reads than writes, approximately 30 thousands IOPS, and about 300 Mbps random read and write rates. Such loads last from 12h to 72h non-stop. The most appropriate benchmark would be SQLite with about 500-2000 simultaneously updatable databases of individual sizes from 10MB to 10GB sizes.
The MINIMUM requirement is to beat standard BCache setup which we did ourselves. If margin on outperformance for our typical load is large - we can offer extra bonus pay (-;
The combined disk array has to be as large as possible (min. 30TB) and as fast as possible both for READ IOPS and write throughput. This is for high-performance testing setup and not for critical production systems, so possible are not-to-frequent crashes or even some occasional data loss is ACCEPTABLE, if brings performance.
File level solutions like standard configuration of `BCache`does not handle out needs well. Because the usage on this array is multiple large database files (append-only and indexes R/W) or very large virtualization files in which VMs are run. So it is database-like intensive and random load. Secondary type of load is heavy duty software compilation and linking.
The typical load is 6-10x more reads than writes, approximately 30 thousands IOPS, and about 300 Mbps random read and write rates. Such loads last from 12h to 72h non-stop. The most appropriate benchmark would be SQLite with about 500-2000 simultaneously updatable databases of individual sizes from 10MB to 10GB sizes.
The MINIMUM requirement is to beat standard BCache setup which we did ourselves. If margin on outperformance for our typical load is large - we can offer extra bonus pay (-;