Latency is 56.875 usec - Задержка в сети
Linstor
fio -ioengine=libaio -direct=1 -name=test -bs=4M -iodepth=16 -rw=write -runtime=60 -filename=/dev/drbd/by-res/vm-300-disk-2/0
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=16
fio-3.12
Starting 1 process
Jobs: 1 (f=0): [f(1)][100.0%][w=60.1MiB/s][w=15 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=42983: Wed Jan 13 13:51:14 2021
 write: IOPS=13, BW=55.4MiB/s (58.1MB/s)(3416MiB/61633msec); 0 zone resets
   slat (usec): min=264, max=9536, avg=558.38, stdev=321.16
   clat (msec): min=19, max=3411, avg=1152.09, stdev=763.17
    lat (msec): min=20, max=3411, avg=1152.65, stdev=763.20
   clat percentiles (msec):
    |  1.00th=[  279],  5.00th=[  609], 10.00th=[  634], 20.00th=[  651],
    | 30.00th=[  676], 40.00th=[  709], 50.00th=[  751], 60.00th=[  810],
    | 70.00th=[ 1099], 80.00th=[ 1905], 90.00th=[ 2534], 95.00th=[ 2769],
    | 99.00th=[ 3037], 99.50th=[ 3205], 99.90th=[ 3406], 99.95th=[ 3406],
    | 99.99th=[ 3406]
  bw (  KiB/s): min= 8192, max=180224, per=100.00%, avg=61355.42, stdev=36030.41, samples=112
  iops        : min=    2, max=   44, avg=14.90, stdev= 8.78, samples=112
 lat (msec)   : 20=0.12%, 100=0.12%, 250=0.70%, 500=2.58%, 750=46.60%
 lat (msec)   : 1000=18.85%
 cpu          : usr=0.53%, sys=0.30%, ctx=843, majf=0, minf=13
 IO depths    : 1=0.1%, 2=0.2%, 4=0.5%, 8=0.9%, 16=98.2%, 32=0.0%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
    issued rwts: total=0,854,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 WRITE: bw=55.4MiB/s (58.1MB/s), 55.4MiB/s-55.4MiB/s (58.1MB/s-58.1MB/s), io=3416MiB (3582MB), run=61633-61633msec

CEPH

fio -ioengine=rbd -direct=1 -name=test -bs=4M -iodepth=16 -rw=write -pool=rbdstorage -runtime=60 -rbdname=image02
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=rbd, iodepth=16
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][3.9%][w=4100KiB/s][w=1 IOPS][eta 25m:21s]
test: (groupid=0, jobs=1): err= 0: pid=73482: Fri Jan 15 12:44:40 2021
 write: IOPS=16, BW=65.6MiB/s (68.8MB/s)(4008MiB/61100msec); 0 zone resets
   slat (usec): min=1085, max=16085, avg=2423.95, stdev=1544.70
   clat (msec): min=171, max=1944, avg=971.59, stdev=215.76
    lat (msec): min=175, max=1946, avg=974.02, stdev=215.64
   clat percentiles (msec):
    |  1.00th=[  514],  5.00th=[  659], 10.00th=[  718], 20.00th=[  793],
    | 30.00th=[  852], 40.00th=[  911], 50.00th=[  969], 60.00th=[ 1020],
    | 70.00th=[ 1083], 80.00th=[ 1133], 90.00th=[ 1234], 95.00th=[ 1318],
    | 99.00th=[ 1552], 99.50th=[ 1754], 99.90th=[ 1921], 99.95th=[ 1938],
    | 99.99th=[ 1938]
  bw (  KiB/s): min= 8192, max=131072, per=99.47%, avg=66817.45, stdev=19644.53, samples=121
  iops        : min=    2, max=   32, avg=16.27, stdev= 4.82, samples=121
 lat (msec)   : 250=0.10%, 500=0.90%, 750=13.47%, 1000=41.82%
 cpu          : usr=3.12%, sys=0.93%, ctx=1005, majf=0, minf=171126
 IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=98.5%, 32=0.0%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
    issued rwts: total=0,1002,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 WRITE: bw=65.6MiB/s (68.8MB/s), 65.6MiB/s-65.6MiB/s (68.8MB/s-68.8MB/s), io=4008MiB (4203MB), run=61100-61100msec

Disk stats (read/write):
   dm-1: ios=0/1246, merge=0/0, ticks=0/9108, in_queue=9108, util=13.43%, aggrios=36/750, aggrmerge=0/496, aggrticks=1609/7501, aggrin_queue=8556, aggrutil=13.67%
 sda: ios=36/750, merge=0/496, ticks=1609/7501, in_queue=8556, util=13.67%

4M, что ты тестишь?

4M, что ты тестишь?

4k 128 надо?

Щас я проснусь и напишу как надо))

Щас я проснусь и напишу как надо))

Linstor
fio -ioengine=libaio -direct=1 -sync=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60 -filename=/dev/drbd/by-res/vm-300-disk-2/0
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=32KiB/s][w=8 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=31559: Wed Jan 13 13:53:36 2021
 write: IOPS=26, BW=105KiB/s (107kB/s)(6296KiB/60019msec); 0 zone resets
   slat (usec): min=25, max=196548, avg=540.91, stdev=5217.58
   clat (usec): min=4, max=363980, avg=37573.36, stdev=35873.21
    lat (msec): min=6, max=364, avg=38.12, stdev=35.77
   clat percentiles (usec):
    |  1.00th=[   174],  5.00th=[  7242], 10.00th=[ 24773], 20.00th=[ 25035],
    | 30.00th=[ 25035], 40.00th=[ 25035], 50.00th=[ 26608], 60.00th=[ 33162],
    | 70.00th=[ 37487], 80.00th=[ 41681], 90.00th=[ 50070], 95.00th=[ 74974],
    | 99.00th=[227541], 99.50th=[250610], 99.90th=[337642], 99.95th=[362808],
    | 99.99th=[362808]
  bw (  KiB/s): min=   24, max=  168, per=100.00%, avg=104.78, stdev=33.82, samples=120
  iops        : min=    6, max=   42, avg=26.13, stdev= 8.46, samples=120
 lat (usec)   : 10=0.64%, 50=0.06%, 100=0.06%, 250=0.32%, 500=0.25%
 lat (usec)   : 750=0.51%, 1000=0.32%
 lat (msec)   : 2=0.64%, 4=1.27%, 10=1.21%, 20=0.06%, 50=84.56%
 lat (msec)   : 100=6.10%, 250=3.56%, 500=0.44%
 cpu          : usr=0.08%, sys=0.17%, ctx=1742, majf=0, minf=9
 IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    issued rwts: total=0,1574,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
 WRITE: bw=105KiB/s (107kB/s), 105KiB/s-105KiB/s (107kB/s-107kB/s), io=6296KiB (6447kB), run=60019-60019msec

CEPH
fio -ioengine=rbd -direct=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -pool=rbdstorage -runtime=60 -rbdname=image02
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=80KiB/s][w=20 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=74479: Fri Jan 15 12:47:11 2021
 write: IOPS=15, BW=62.0KiB/s (63.5kB/s)(3724KiB/60039msec); 0 zone resets
   slat (nsec): min=10065, max=56318, avg=26527.20, stdev=4085.50
   clat (msec): min=14, max=516, avg=64.45, stdev=46.69
    lat (msec): min=14, max=516, avg=64.48, stdev=46.69
   clat percentiles (msec):
    |  1.00th=[   18],  5.00th=[   32], 10.00th=[   34], 20.00th=[   39],
    | 30.00th=[   41], 40.00th=[   43], 50.00th=[   50], 60.00th=[   59],
    | 70.00th=[   74], 80.00th=[   81], 90.00th=[  101], 95.00th=[  136],
    | 99.00th=[  271], 99.50th=[  326], 99.90th=[  518], 99.95th=[  518],
    | 99.99th=[  518]
  bw (  KiB/s): min=    8, max=  104, per=99.84%, avg=61.90, stdev=16.27, samples=120
  iops        : min=    2, max=   26, avg=15.40, stdev= 4.07, samples=120
 lat (msec)   : 20=1.83%, 50=49.09%, 100=38.88%, 250=8.70%, 500=1.40%
 lat (msec)   : 750=0.11%
 cpu          : usr=0.09%, sys=0.04%, ctx=932, majf=0, minf=4
 IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    issued rwts: total=0,931,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
 WRITE: bw=62.0KiB/s (63.5kB/s), 62.0KiB/s-62.0KiB/s (63.5kB/s-63.5kB/s), io=3724KiB (3813kB), run=60039-60039msec

4k 128 надо?

26 иопс vs 15 иопс. хахаха