Disk Performance

Main Factors Affecting Performance

• Total number of disks on storage nodes - more disks result in better performance
• Memory of storage nodes - more memory allows for more caching
• Gigabit network will definitely become a bottleneck; at minimum ensure 10 Gigabit network

Cluster Configuration

Since storage systems vary greatly across clusters, the performance tests here are based on a cluster currently in use, with the following configuration:

• 4 storage nodes
• Two storage nodes with 22 HDD disks
• Two storage nodes with 24 NVME SSD disks
• Internal network is 100 Gbps Ethernet

Concept Explanation

IO Depth (iodepth): The maximum number of "in-flight" requests for a test process (i.e., requests that have been submitted but not yet completed). The test process is not allowed to continue submitting requests when the number of in-flight requests exceeds the IO depth.

Number of Processes (procs): The number of processes running tests simultaneously.

Read/Write Mode (mode): read indicates reading, write indicates writing, rw indicates alternating read and write. The rand prefix indicates random read/write; otherwise it's sequential read/write.

Block Size (block size, bs): The size of the data block for a single IO request. 4k appears in read/write configuration files, when reading/writing small fragmented files. 4M appears when reading/writing large compressed files, video files, etc.

IOPS: IO per second, the number of requests processed per second.

Throughput: The amount of data read/written per second

Temporary Storage Space Performance Test Command

fio --name=disktest \
    --ioengine=libaio \
    --iodepth=$IODEPTH \
    --numjobs=$PROCS \
    --rw=$MODE \
    --bs=$BLOCK_SIZE \
    --direct=1 --buffered=0 \
    --size=2G \
    --runtime=30 \
    --time_based \
    --group_reporting \
    --output-format=json

Dataset Performance Test Command

fio --name=disktest \
    --ioengine=libaio \
    --iodepth=$IODEPTH \
    --numjobs=$PROCS \
    --rw=$MODE \
    --bs=$BLOCK_SIZE \
    --direct=1 --buffered=0 \
    --filename=/input0/train.zip \   # A 14G file
    --runtime=30 \
    --time_based \
    --group_reporting \
    --output-format=json