Fio

From Omnia
Revision as of 22:14, 14 December 2022 by Kenneth (talk | contribs) (→‎Source method)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

fio - Flexible IO tester

fio - Flexible IO tester (FIO) benchmarking and workload simulation tool

-

fio - Flexible IO Tester - http://git.kernel.dk/?p=fio.git

-

git.kernel.dk Git - http://git.kernel.dk/?p=fio.git

description: fio - Flexible IO Tester
owner: Jens Axboe

-

Jens Axboe - Wikipedia - http://en.wikipedia.org/wiki/Jens_Axboe

Jens Axboe is a Linux kernel hacker. He is the current Linux kernel maintainer of the block layer and other block devices, along with contributing the CFQ I/O scheduler, Noop scheduler, Deadline scheduler and splice (system call) IO architecture. Jens is also the author of the blktrace utility and kernel parts, which provides a way to trace every block IO activity in the Linux kernel. blktrace exists in 2.6.17 and later Linux kernels.
To facilitate his block layer work in the Linux kernel Jens created the Flexible IO tester (FIO) benchmarking and workload simulation tool. FIO is able to simulate various types of I/O loads, such as synchronous, asynchronous, mmap, etc., as well as specifying the number of threads or processes, read vs. write mix, and various other parameters. FIO was used to set the world record in April 2009 for the highest number of I/Os-per-second (IOPS) in a single system

Disambiguation

See also: Fusion-io (FIO)

Source

git clone git://git.kernel.dk/fio.git

or 

git clone https://github.com/axboe/fio.git

Usage

# 12 hour write test
fio --name=write_job --filename=/dev/nvme0n1 --iodepth=128 --bs=256k --rw=write --direct=1 --runtime=12h --time_based --ioengine=libaio

# 5 minute write test
fio --name=write_job --filename=/dev/nvme0n1 --iodepth=128 --bs=256k --rw=write --direct=1 --runtime=5m --time_based --ioengine=libaio

# 1 minute write test
fio --name=write_job --filename=/dev/nvme0n1 --iodepth=128 --bs=256k --rw=write --direct=1 --runtime=1m --time_based --ioengine=libaio

Installation

fio - Flexible IO Tester - http://git.kernel.dk/?p=fio.git

Source method

Source build and install: 

# RHEL DEPENDENCIES
sudo yum -y install gcc make wget git libaio-devel zlib-devel
# may require # yum -y --enablerepo=rpmforge install git

# UBUNTU DEPENDENCIES
sudo apt-get install gcc make wget git libaio-dev zlib1g-dev

# MAKE SURE LIBAIO IS INCLUDED, OR PERFORMANCE WILL BE TERRIBLE!

# BUILD
mkdir -p ~/.src ; cd ~/.src
git clone git://git.kernel.dk/fio.git
cd fio
# git checkout fio-2.0.9
# git checkout fio-2.16
# git checkout fio-3.0
# git checkout fio-3.12
git checkout fio-3.16
# git checkout fio-3.19  # newer option
./configure
make clean
make

# INSTALL
cp fio /usr/local/bin
# OR this will include the man and fio_generate_plots page as well
sudo make install

RPMForge method

RPMForge fio install: 

yum -y --enablerepo=rpmforge install fio

Freshmeat method

fio utility - http://freshmeat.net/projects/fio 

yum -y install gcc libaio-devel make wget
mkdir -p ~/src ; cd ~/src
wget "http://freshmeat.net/urls/3aa21b8c106cab742bf1f20d60629e3f"
tar -zvxf fio-1.57.tar.gz
cd fio-1.57
make

cp fio /usr/local/bin
# OR
make install

Windows Installation

Windows fio - flexible io tester - http://bluestop.org/fio/

"This site contains Windows binaries for fio. The main fio site, which conatins the latest code, is at http://git.kernel.dk/?p=fio.git."

FIO sources which were used to build the current installer can be found in fio.zip.
The Windows version of FIO uses Cygwin.
Sources for the Cygwin binaries which are used in the current installer can be found in cygwin_src.zip.
http://bluestop.org/fio/cygwin_src.zip

Fio is installed to: 

C:\Program Files\fio\fio.exe

Ported to Windows by Bruce Cran <BCran@fusionio.com>.

Execution: 

"C:\Program Files\fio\fio.exe" --name=job1 --filename=\\.\PhysicalDrive1 --size=100m --rw=write --thread

Windows threading sucks! 

fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.

Errors you will see when the drive fills: 

fio: io_u error on file \\.\PhysicalDrive1: Input/output error: read offset=1073737728, buflen=4096
fio: pid=7980, err=5/file:io_u.c:1603, func=io_u error, error=Input/output error

job1: (groupid=0, jobs=1): err= 5 (file:io_u.c:1603, func=io_u error, error=Input/output error): pid=7980: Wed Apr 6 20:

VMware ESXi

Support for ESXi is minimal. 

commit c89318761586dbd77b8f31ce0ed68ff4fc086c89
Author: Jens Axboe <axboe@fb.com>
Date:   Wed Jun 18 15:30:09 2014 -0700

    Add support for compiling for ESX

    With contributions from Ryan Haynes <rhaynes@fusionio.com>

    Signed-off-by: Jens Axboe <axboe@fb.com>

From within a ESXi ddk build environment: (will need a separate build for 5.x and 6.x) 

# sinter esxi5
mkdir -p ~/.src ; cd ~/.src
git clone git://git.kernel.dk/fio.git
cd fio
git checkout c89318761586dbd77b8f31ce0ed68ff4fc086c89

# build
make clean
./configure --esx
make

# verify linked
ldd fio

# copy to test server
scp fio root@vmware-server:/scratch/
ssh root@vmware-server

# test
cd /scratch
./fio --help

# debug
cd /scratch
strace ./fio

# quick write test
./fio --name=job1 --filename=/vmfs/volumes/some_datastore/test.txt --size=1m --rw=write
# quick read test
./fio --name=job1 --filename=/vmfs/volumes/some_datastore/test.txt --size=1m

GitHub Issue:

Note: A ESXi 6.0/5.5 build will not work on ESXi 5.0/5.1, but the reverse (5.0 on 6.0) does appear to work. 

./fio: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by ./fio)
./fio: /lib64/libc.so.6: version `GLIBC_2.10' not found (required by ./fio)
./fio: /lib64/libc.so.6: version `GLIBC_2.6' not found (required by ./fio)

Fusion-io Compiling the fio Utility

The fio benchmarking utility is used to verify Linux system performance with an ioDrive. To compile the fio utility, perform the following steps

Grab the latest version of fio source from the tarball link in http://freshmeat.net/projects/fio

Install the necessary standard dependencies. For example: 

$ yum -y install gcc

Install the necessary libaio development headers. For example: 

$ yum -y install libaio-devel

Explode the fio tarball: 

$ tar xjvf fio-X.Y.Z.tar.bz2   
$ cd fio-X.Y.Z

Build fio for your system: 

$ make

When the rebuild completes successfully, a fio binary is placed in the fio directory.

Job Options

Job Options: 

name=str	# job name (if 'global' apply to all jobs)
filename=str	# target file or drive (/dev/sdb)
readwrite=str	# (rw=str) traffic type: read, write, randread, randwrite, rw/readwrite, randrw
size=int	# size for job (1k, 1m, 1g, 20%, etc)
blocksize=int	# block size: 4k (default), 1k, 1m, 1g, etc
ioengine=str	# io engine: sync (default), libaio (linux native asyncrhonous io), etc
iodepth=int	# I/O queue size
direct=bool	# use non-buffered I/O (default: 0 - false)
thinktime=int	# stall job microseconds between issuing I/Os
nice=int	# run job at this nice value
cpuload=int	# limit percenage of CPU cycles
runtime=int	# number of seconds to run
loops=int	# interations for job (default: 1)
numjobs=int	# number of clones of job (default: 1)

Basic Traffic Tests

Simple read job: 

fio --name=job1 --filename=/dev/sdb
fio --name=job1 --filename=/dev/sdb --size=10g

Simple write job: 

fio --name=job1 --filename=/dev/sdb --size=10g --rw=write

Simple read job: 

fio --name=job1 --filename=/dev/sdb --size=10g --rw=read

Simple two process write job: 

fio --name=global --filename=/dev/sdb --size=10g --rw=write --name=job1 --name=job2

Complex Write random traffic to /dev/sdb: 

fio --name=random-writers --ioengine=libaio --iodepth=1 --rw=randwrite --bs=32k --direct=0 --numjobs=1 --loops=2 --filename=/dev/sdb

High bandwidth job: 

fio --name=job1 --ioengine=libaio --iodepth=4 --rw=write --bs=1m --direct=1 --numjobs=1 --size=30g --filename=/dev/sdb

Unaligned DMA: (FreeBSD) 

fio --name=job1 --filename=/dev/ad1 --size=5g --iomem_align=1 --rw=write --bs=256k --thread

Random read job and write job: 

fio --name=global --ioengine=libaio --iodepth=4 --bs=4k --direct=0 --loops=2 --filename=/dev/sdb --name=randread --rw=randread --name=randwrite --rw=randwrite

Basic Report

Read Report: 

# fio --filename=/dev/sdb --name=job1 --size=10g
job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.11-46-g5a90
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [623.4M/0K/0K /s] [160K/0 /0  iops] [eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=1895: Sat Dec  8 23:20:13 2012
  read : io=10240MB, bw=637471KB/s, iops=159367 , runt= 16449msec
    clat (usec): min=0 , max=2083 , avg= 5.78, stdev=47.34
     lat (usec): min=0 , max=2083 , avg= 5.86, stdev=47.35
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    1], 20.00th=[    1],
     | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[  106], 99.50th=[  532], 99.90th=[  564], 99.95th=[  604],
     | 99.99th=[  812]
    bw (KB/s)  : min=597504, max=655360, per=100.00%, avg=637531.25, stdev=10370.41
    lat (usec) : 2=96.31%, 4=2.01%, 10=0.01%, 20=0.09%, 50=0.02%
    lat (usec) : 100=0.52%, 250=0.32%, 500=0.15%, 750=0.55%, 1000=0.04%
    lat (msec) : 2=0.01%, 4=0.01%
  cpu          : usr=8.70%, sys=40.61%, ctx=22011, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2621440/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=10240MB, aggrb=637470KB/s, minb=637470KB/s, maxb=637470KB/s, mint=16449msec, maxt=16449msec

Disk stats (read/write):
  sdb: ios=42796/0, merge=2575677/0, ticks=26606/0, in_queue=26577, util=81.95%

Write Report: 

# fio --filename=/dev/sdb --name=job1 --size=10g --rw=write
job1: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.11-46-g5a90
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0K/163.5M/0K /s] [0 /41.9K/0  iops] [eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=1902: Sat Dec  8 23:22:21 2012
  write: io=10240MB, bw=304456KB/s, iops=76113 , runt= 34441msec
    clat (usec): min=1 , max=926829 , avg=12.00, stdev=815.23
     lat (usec): min=2 , max=926829 , avg=12.19, stdev=815.23
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    6], 95.00th=[   10],
     | 99.00th=[   18], 99.50th=[   32], 99.90th=[ 1736], 99.95th=[ 6048],
     | 99.99th=[14656]
    bw (KB/s)  : min=  132, max=741784, per=100.00%, avg=312652.71, stdev=74487.53
    lat (usec) : 2=0.01%, 4=59.53%, 10=35.16%, 20=4.46%, 50=0.54%
    lat (usec) : 100=0.04%, 250=0.11%, 500=0.05%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.04%, 10=0.03%, 20=0.02%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=11.07%, sys=30.73%, ctx=7795, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=2621440/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=10240MB, aggrb=304455KB/s, minb=304455KB/s, maxb=304455KB/s, mint=34441msec, maxt=34441msec

Disk stats (read/write):
  sdb: ios=83/19960, merge=0/2541727, ticks=5/4637755, in_queue=4655340, util=99.35%

Benchmark

simple job file: defines two processes, each randomly reading from a 128MB file: 

[global]
rw=randread
size=128m

[job1]

[job2]

command line version: 

fio --name=global --rw=randread --size=128m --name=job1 --name=job2

---

4 processes each randomly writing to their own files:

  • We want to use async io here
  • depth of 4 for each file
  • increased the buffer size used to 32KB
  • numjobs to 4 to fork 4 identical jobs
  • 64MB file
[random-writers]
ioengine=libaio
iodepth=4
rw=randwrite
bs=32k
direct=0
size=64m
numjobs=4

command line version: 

fio --name=random-writers --ioengine=libaio --iodepth=4 --rw=randwrite --bs=32k --direct=0 --size=64m --numjobs=4

---

Fusion-io: fio random 512 read peak IOPS: (Run for 150 seconds (120 + 30) or 2.5 minutes) 

[global]
readwrite=randrw
rwmixread=100
blocksize=512
ioengine=libaio
numjobs=4
thread=1
direct=1
iodepth=32
iodepth_batch=16
iodepth_batch_complete=16
group_reporting=1
ramp_time=30
norandommap=1
description=fio random 512 read peak IOPS
time_based=1
runtime=120

[/dev/sdb]
filename=/dev/sdb

Command: fio --minimal --output=fio.output fio-job.ini

Inspecting disk IO performance with fio

Inspecting disk IO performance with fio | Linux.com - https://www.linux.com/learn/tutorials/442451-inspecting-disk-io-performance-with-fio

Linux.com has published article about a new tool called fio:

fio was created to allow benchmarking specific disk IO workloads. It can issue its IO requests using one of many synchronous and asynchronous IO APIs, and can also use various APIs which allow many IO requests to be issued with a single API call. You can also tune how large the files fio uses are, at what offsets in those files IO is to happen at, how much delay if any there is between issuing IO requests, and what if any filesystem sync calls are issued between each IO request. A sync call tells the operating system to make sure that any information that is cached in memory has been saved to disk and can thus introduce a significant delay. The options to fio allow you to issue very precisely defined IO patterns and see how long it takes your disk subsystem to complete these tasks.

random-read-test.fio

"The first test you might like to perform is for random read IO performance. This is one of the nastiest IO loads that can be issued to a disk, because it causes the disk head to seek a lot, and disk head seeks are extremely slow operations relative to other hard disk operations. One area where random disk seeks can be issued in real applications is during application startup, when files are requested from all over the hard disk. You specify fio benchmarks using configuration files with an ini file format. You need only a few parameters to get started. rw=randread tells fio to use a random reading access pattern, size=128m specifies that it should transfer a total of 128 megabytes of data before calling the test complete, and the directory parameter explicitly tells fio what filesystem to use for the IO benchmark. On my test machine, the /tmp filesystem is an ext3 filesystem stored on a RAID-5 array consisting of three 500GB Samsung SATA disks. If you don't specify directory, fio uses the current directory that the shell is in, which might not be what you want. "

random-read-test.fio: 

; random read of 128mb of data

[random-read]
rw=randread
size=128m
directory=/tmp/fio-testing/data

Test execution: 

$ fio random-read-test.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
Starting 1 process
random-read: Laying out IO file(s) (1 file(s) / 128MiB)
Jobs: 1 (f=1): [r] [100.0% done] [  3588/     0 kb/s] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=30598
  read : io=128MiB, bw=864KiB/s, iops=211, runt=155282msec
    clat (usec): min=139, max=148K, avg=4736.28, stdev=6001.02
    bw (KiB/s) : min=  227, max= 5275, per=100.12%, avg=865.00, stdev=362.99
  cpu          : usr=0.07%, sys=1.27%, ctx=32783, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     issued r/w: total=32768/0, short=0/0
     lat (usec): 250=34.92%, 500=0.36%, 750=0.02%, 1000=0.05%
     lat (msec): 2=0.41%, 4=12.80%, 10=44.96%, 20=5.16%, 50=0.94%
     lat (msec): 100=0.37%, 250=0.01%

Run status group 0 (all jobs):
   READ: io=128MiB, aggrb=864KiB/s, minb=864KiB/s, maxb=864KiB/s, mint=155282msec, maxt=155282msec

Disk stats (read/write):
  dm-6: ios=32768/148, merge=0/0, ticks=154728/12490, in_queue=167218, util=99.59%


random-read-test-aio.fio

random-read-test-aio.fio: 

; random read of 128mb of data with libaio

[random-read]
rw=randread
size=128m
directory=/tmp/fio-testing/data

ioengine=libaio
iodepth=8
direct=1
invalidate=1

Execution: 

$ fio random-read-test-aio.fio
random-read: (groupid=0, jobs=1): err= 0: pid=31318
  read : io=128MiB, bw=2,352KiB/s, iops=574, runt= 57061msec
    slat (usec): min=8, max=260, avg=25.90, stdev=23.23
    clat (usec): min=1, max=124K, avg=13901.91, stdev=12193.87
    bw (KiB/s) : min=    0, max= 5603, per=97.59%, avg=2295.43, stdev=590.60
...
  IO depths    : 1=0.1%, 2=0.1%, 4=4.0%, 8=96.0%, 16=0.0%, 32=0.0%, >=64=0.0%
...
Run status group 0 (all jobs):
   READ: io=128MiB, aggrb=2,352KiB/s, minb=2,352KiB/s, maxb=2,352KiB/s, mint=57061msec, maxt=57061msec

high load test

fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
  --loops=10000 --size=9654901800 --runtime=600 --rwmixwrite=100 --do_verify=0 --filename=/dev/fioa --thread

fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
  --loops=10000 --size=9654901800 --runtime=600 --rwmixwrite=100 --do_verify=0 --filename=/dev/fiob --thread

Source: Fusion thermal monitoring test

Mix: (adjust rwmixwrite) 

fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
  --loops=10000 --size=9654901800 --runtime=600 --rwmixwrite=60 --do_verify=0 --filename=/dev/fioa --thread

10 minute load

100% Random Write: 

# WRITE: bw=1381MiB/s (1448MB/s), 21.6MiB/s-21.6MiB/s (22.6MB/s-22.6MB/s), io=809GiB (869GB), run=600020-600186msec
#
fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
  --loops=10000 --runtime=600 --rwmixwrite=100 --filename=/dev/fioa --thread --allow_mounted_write=1

# WRITE: bw=1992MiB/s (2088MB/s), 25.5MiB/s-93.6MiB/s (26.7MB/s-98.2MB/s), io=1167GiB (1253GB), run=600022-600158msec
#
fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
  --loops=10000 --runtime=600 --rwmixwrite=100 --filename=/dev/nvme0n1 --thread

80% Random Write / 20% Random Read: 

# READ: bw=291MiB/s (305MB/s), 4611KiB/s-4720KiB/s (4722kB/s-4834kB/s), io=170GiB (183GB), run=600181-600195msec
# WRITE: bw=1164MiB/s (1221MB/s), 18.2MiB/s-18.2MiB/s (19.0MB/s-19.1MB/s), io=682GiB (733GB), run=600181-600195msec
#
fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
  --loops=10000 --runtime=600 --rwmixwrite=80 --filename=/dev/fioa --thread --allow_mounted_write=1

# READ: bw=446MiB/s (468MB/s), 5900KiB/s-10.2MiB/s (6041kB/s-10.7MB/s), io=261GiB (281GB), run=600014-600163msec
# WRITE: bw=1785MiB/s (1872MB/s), 23.2MiB/s-40.8MiB/s (24.3MB/s-42.8MB/s), io=1046GiB (1123GB), run=600014-600163msec
#
fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
  --loops=10000 --runtime=600 --rwmixwrite=80 --filename=/dev/nvme0n1 --thread

New ETA

Do not report live ETA: 

fio --eta=never ...

Trim

fio can issue trim command: 

fio --name=test --random_generator=lfsr --ioengine=sg --rw=randtrim --filename=/dev/sg4 --iodepth=8 --iodepth_batch=8 --loops=3

Minimal Output

Fio minimal output field indexes - https://www.andypeace.com/fio_minimal.html

"fio, the flexible IO tester, is a very useful tool for benchmarking IO performance. It has an option to produce minimal output, which is very useful when gathering data for later processing, e.g. graphing. The man page for fio describes the output format, but does not number the fields. This means that when extracting fields, one must count the fields in the man page to find the correct index to extract." 

Field	Description
1	terse version
2	fio version
3	jobname
4	groupid
5	error
Read status:
6	Total I/O (KB)
7	bandwidth (KB/s)
8	IOPS
9	runtime (ms)
Submission latency:
10	min
11	max
12	mean
13	standard deviation
Completion latency:
14	min
15	max
16	mean
17	standard deviation
Completion latency percentiles (20 fields):
18-37	Xth percentile=usec
Total latency:
38	min
39	max
40	mean
41	standard deviation
Bandwidth:
42	min
43	max
44	aggregate percentage of total
45	mean
46	standard deviation
Write status:
47	Total I/O (KB)
48	bandwidth (KB/s)
49	IOPS
50	runtime (ms)
Submission latency:
51	min
52	max
53	mean
54	standard deviation
Completion latency:
55	min
56	max
57	mean
58	standard deviation
Completion latency percentiles (20 fields):
59-78	Xth percentile=usec
Total latency:
79	min
80	max
81	mean
82	standard deviation
Bandwidth:
83	min
84	max
85	aggregate percentage of total
86	mean
87	standard deviation
CPU usage:
88	user
89	system
90	context switches
91	major page faults
92	minor page faults
IO depth distribution:
93	<=1
94	2
95	4
96	8
97	16
98	32
99	>=64
IO latency distribution (microseconds):
100	<=2
101	4
102	10
103	20
104	50
105	100
106	250
107	500
108	750
109	1000
IO latency distribution (milliseconds):
110	<=2
111	4
112	10
113	20
114	50
115	100
116	250
117	500
118	750
118	1000
120	2000
121	>=2000
Disk utilization (1 for each disk used, for disk n, n is zero-based):
122+10n	name
123+10n	read ios
124+10n	write ios
125+10n	read merges
126+10n	write merges
127+10n	read ticks
128+10n	write ticks
129+10n	read in-queue time
130+10n	write in-queue time
131+10n	disk utilization
132+10n	percentage
Error Info (dependent on continue_on_error, default off):
F-1	total # errors
F-0	first error code
newline	text description (if provided in config - appears on newline)

Verification Testing

verify=str
If writing to a file, fio can verify the file contents after each iteration of the job. Each verification method also implies verification of special header, which is written to the beginning of each block. This header also includes meta information, like offset of the block, block number, timestamp when block was written, etc. verify can be combined with verify_pattern option.

See https://fio.readthedocs.io/en/latest/fio_doc.html#verification

Samples

Running a read zero job: (Kep K.) 

[global]
readwrite=randrw
rwmixread=100
blocksize=512
ioengine=libaio
numjobs=1
thread=0
direct=1
iodepth=32
iodepth_batch=16
iodepth_batch_complete=16
group_reporting=1
#ramp_time=5
norandommap=1
description=fio random 512 read zero IOPS
time_based=1
runtime=30
randrepeat=0


[/dev/fioa]
filename=/dev/fioa
cpus_allowed=1
startdelay=1

[/dev/fioa]
filename=/dev/fioa
cpus_allowed=2
startdelay=2

[/dev/fioa]
filename=/dev/fioa
cpus_allowed=3
startdelay=3

[/dev/fioa]
filename=/dev/fioa
cpus_allowed=4
startdelay=4

Linked Clones Example

by Kenneth Burgener 

[random-writers]
ioengine=libaio
iodepth=1
rw=randwrite
bs=32k
direct=0
; nice=19
; thinktime=10000
; size=1g
numjobs=1
loops=99999
; cpuload=10
filename=/dev/sdb1

Command line version: 

fio --name=random-writers --ioengine=libaio --iodepth=1 --rw=randwrite --bs=32k --direct=0 --numjobs=1 --loops=99999 --filename=/dev/sdb

Bandwidth Test

[global]
readwrite=randrw
rwmixread=0
blocksize=1M
ioengine=libaio
numjobs=4
thread=0
direct=1
iodepth=32
iodepth_batch=16
iodepth_batch_complete=16
group_reporting=1
ramp_time=5
norandommap=1
description=fio random 1M write peak BW
time_based=1
runtime=60
randrepeat=0

[/dev/sdb]
filename=/dev/sdb

Only changing the blocksize in the following fio config: 

|8K|  WRITE: io=24378MB, aggrb=416040KB/s, minb=426025KB/s, 
|8K| maxb=426025KB/s, mint=60002msec, maxt=60002msec|
|16K|  WRITE: io=37994MB, aggrb=648328KB/s, minb=663888KB/s, 
|16K| maxb=663888KB/s, mint=60009msec, maxt=60009msec|
|32K|  WRITE: io=50756MB, aggrb=866155KB/s, minb=886943KB/s, 
|32K| maxb=886943KB/s, mint=60006msec, maxt=60006msec|
|64K|  WRITE: io=60554MB, aggrb=1009.5MB/s, minb=1033.3MB/s, 
|64K| maxb=1033.3MB/s, mint=60011msec, maxt=60011msec|
|128K|  WRITE: io=18419MB, aggrb=314246KB/s, minb=321788KB/s, 
|128K| maxb=321788KB/s, mint=60020msec, maxt=60020msec|
|256K|  WRITE: io=21555MB, aggrb=366735KB/s, minb=375536KB/s, 
|256K| maxb=375536KB/s, mint=60186msec, maxt=60186msec|
|512K|  WRITE: io=29526MB, aggrb=502411KB/s, minb=514469KB/s, 
|512K| maxb=514469KB/s, mint=60179msec, maxt=60179msec|
|1M|  WRITE: io=38220MB, aggrb=649344KB/s, minb=664928KB/s, 
|1M| maxb=664928KB/s, mint=60272msec, maxt=60272msec|
|2M|  WRITE: io=34552MB, aggrb=585724KB/s, minb=599781KB/s, 
|2M| maxb=599781KB/s, mint=60406msec, maxt=60406msec|
|4M|  WRITE: io=34992MB, aggrb=588999KB/s, minb=603135KB/s, 
|4M| maxb=603135KB/s, mint=60835msec, maxt=60835msec|

Another run involves reducing these: 

iodepth=16 iodepth_batch=8 iodepth_batch_complete=8


Windows Sample

Thane H.: Here is a sample ini file that uses raw devices: 

[global]
rw=read
name=fire
verify=crc32c
bs=4096b
iodepth=256
size=64g
thread
[thread_fct0-0]
filename=\\.\PhysicalDrive1
[thread_fct0-1]
filename=\\.\PhysicalDrive1
[thread_fct2-0]
filename=\\.\PhysicalDrive0
[thread_fct2-1]
filename=\\.\PhysicalDrive0

Another sample: 

# fio --name=job1 --filename=e: --size=5g --rw=write --bs=256k --thread

List devices: 

wmic diskdrive list brief

Unaligned DMA

iomem_align=int This indiciates the memory alignment of the IO memory
                buffers. Note that the given alignment is applied to
                the first IO unit buffer, if using iodepth the
                alignment of the following buffers are given by the
                bs used. In other words, if using a bs that is a multiple
                of the page sized in the system, all buffers will be
                aligned to this value. If using a bs that is not page
                aligned, the alignment of subsequent IO memory buffers
                is the sum of the iomem_align and bs used.

Linux: 

fio --name=job1 --filename=/dev/fioa --size=5g --iomem_align=1 --rw=write --bs=256k

FreeBSD: 

fio --name=job1 --filename=/dev/ad1 --size=5g --iomem_align=1 --rw=write --bs=256k --thread
fio --name=job1 --filename=/dev/ad1 --size=1M --iomem_align=1 --direct=1 --rw=write --bs=256k --thread

Windows 7: 

C:\Program Files\fio>fio --name=job1 --filename=e: --size=5g --rw=write --bs=256k --thread --iomem_align=1 --direct=1

Windows 7 Raw Device: (Must run as Administrator) 

C:\Program Files\fio>fio --name=job1 --filename=\\.\PHYSICALDRIVE1 --size=5g --rw=write --bs=256k --thread --iomem_align=1 --direct=1

Misc

fio-4k-r-qd8.ini 

[global]
readwrite=read
rwmixread=100
blocksize=4k
ioengine=aio
numjobs=16
thread=1
direct=1
iodepth=16
iodepth_batch=16
iodepth_batch_complete=16
group_reporting=1
ramp_time=5
norandommap=1
description=fio 4k read IOPS QD 8
time_based=1
runtime=300
randrepeat=0

[/dev/nbd0]
filename=/dev/nbd0

More Examples

Linux: 

fio --name=job1 --filename=/dev/sdb --iodepth=1 --bs=256k --rw=write --direct=1 --runtime=30 --ioengine=libaio

fio --name=job1 --filename=/dev/sdb --iodepth=1 --bs=256k --rw=write --direct=1 --runtime=30

fio --name=job1 --filename=/dev/sdb --iodepth=1 --bs=256k --rw=read --direct=1 --runtime=30

fio --name=job1 --filename=/dev/sdb --iodepth=32 --bs=256k --rw=write --direct=1 --runtime=60

fio --name=job1 --filename=/dev/sdb --iodepth=32 --bs=256k --rw=read --direct=1 --runtime=60

fio --name=job1 --filename=/dev/sdb --iodepth=32 --bs=256k --rw=write --direct=1 --name=job2 --rw=read --filename=/dev/sdb --iodepth=32 --direct=1 --bs=256k

fio --name=job1 --filename=/dev/sdb --iodepth=32 --bs=256k --rw=write --direct=1 --thread --runtime=120

fio --name=job1 --filename=/dev/sdb --iodepth=32 --bs=256k --rw=read --direct=1 --thread --runtime=120

fio --name=global --ioengine=libaio --iodepth=32 --bs=4k --direct=0 --loops=2 --filename=/dev/sdb --name=randread --rw=randread --name=randwrite --rw=randwrite

fio --output=output.txt --filename=/dev/nvme0n1 --iodepth=128 --bs=4k --thread=2 --direct=1 --ioengine=libaio --name=write-phase --rw=write --name=read-phase --rw=read

fio --output=output.txt --filename=/dev/nvme0n1 --iodepth=128 --bs=4k --thread=1 --direct=1 --ioengine=libaio --name=write-phase \
    --do_verify=0 --rw=write --name=read-phase --stonewall --do_verify=0 --time_based=1 --loops=999 --runtime=24h --rw=read

Windows: 

wmic diskdrive list brief

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=1 --bs=256k --rw=write --direct=1 --thread --runtime=30

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=1 --bs=256k --rw=read --direct=1 --thread --runtime=30

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=32 --bs=256k --rw=write --direct=1 --thread

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=32 --bs=256k --rw=read --direct=1 --thread

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=32 --bs=256k --rw=write --direct=1 --name=job2 --rw=read --filename=\\.\PhysicalDrive1 --iodepth=32 --direct=1 --bs=256k --thread

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=32 --bs=256k --rw=write --direct=1 --thread --runtime=120

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=32 --bs=256k --rw=read --direct=1 --thread --runtime=120

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=1 --bs=4k --rw=write --direct=1 --thread --runtime=60

fio --name=global --iodepth=32 --bs=4k --direct=0 --loops=2 --filename=\\.\PhysicalDrive1 --name=randread --rw=randread --name=randwrite --rw=randwrite

nvme random write

fio --name=rand_write_job --filename=/dev/nvme0n1 --iodepth=128 --bs=4k --rw=randwrite --direct=1 --runtime=24h --time_based --ioengine=libaio

fio --name=rand_write_job --filename=/dev/nvme0n1 --iodepth=128 --bs=4k --rw=randwrite --direct=1 --runtime=24h --time_based --ioengine=libaio norandommap random_generator=tausworthe



norandommap random_generator=tausworthe
"Normally fio will cover every block of the file when doing random I/O. If this parameter is given, a new offset will be chosen without looking at past I/O history. This parameter is mutually exclusive with verify."

Output to file statistics

fio --name=global --status-interval=10 --output fio.txt --output-format=terse --eta-newline=10 --numjobs=1 --iodepth=32 --direct=1 --ioengine=libaio --max_latency=5s \
  --norandommap --filename=/dev/sda --name=precondjob --rw=write --bs=128k --fill_device=1

fio --name=global --status-interval=60 --output fio.txt --eta-newline=10 --numjobs=1 --iodepth=32 --direct=1 --ioengine=libaio --max_latency=5s --norandommap \
   --filename=/dev/sda --name=precondjob --rw=write --bs=128k --size=2GB --stonewall --name=readjob --loops=10 --rw=read --bs=128k --stonewall

Report full status every 1 minute: 

--status-interval=1m

fill then read forever

Limit to only first 20GB

fill-read-forever.fio: 

[write]
rw=write
bs=128k
direct=1
ioengine=libaio
iodepth=128
size=20G
filename=/dev/nvme0n1
stonewall

[read]
rw=read
bs=128k
direct=1
ioengine=libaio
iodepth=128
size=20G
filename=/dev/nvme0n1
runtime=9999h
time_based
stonewall

Standard

/dev/sdb
/dev/nvme0n1


Precondition Write (full drive write)

fio --name=global --iodepth=32 --bs=128k --direct=1 --ioengine=libaio \
  --filename=/dev/nvme0n1 --name=precondjob --rw=write --fill_device=1

Sequential Write (time based)

fio --name=global --iodepth=32 --bs=128k --direct=1 --ioengine=libaio \
  --filename=/dev/nvme0n1 --name=writejob --rw=write --runtime=24h --time_based

Precondition: (fill drive) 

fio --name=global --iodepth=32 --bs=128k --direct=1 --ioengine=libaio \
  --filename=/dev/nvme0n1 --name=precondition --rw=write --fill_device=1

Sequential Read

fio --name=global --iodepth=32 --bs=128k --direct=1 --ioengine=libaio \
  --filename=/dev/nvme0n1 --name=readjob --rw=read --runtime=24h --time_based

Sequential Read (full drive read)

fio --name=global --iodepth=32 --bs=128k --direct=1 --ioengine=libaio \
  --filename=/dev/nvme0n1 --name=readjob --rw=read

Standard with Status Interval

Precondition Write

fio --name=global --status-interval=60 --output fio-precond.txt --iodepth=32 --bs=128k --direct=1 \
  --ioengine=libaio --max_latency=5s --norandommap \
  --filename=/dev/sdb --name=precondjob --rw=write --fill_device=1

Sequential Write

fio --name=global --status-interval=60 --output fio-write.txt --iodepth=32 --bs=128k --direct=1 \
  --ioengine=libaio --max_latency=5s --norandommap \
  --filename=/dev/sdb --name=writejob --rw=write --runtime=24h --time_based

With Precondition: 

fio --name=global --status-interval=60 --output fio-precond.txt --iodepth=128 --bs=128k --direct=1 \
  --ioengine=libaio --max_latency=5s --norandommap \
  --filename=/dev/nvme0n1 \
  --name=precondjob --rw=write --fill_device=1 \
  --name=writejob --rw=write --runtime=24h --time_based --stonewall

Sequential Read

fio --name=global --status-interval=60 --output fio-read.txt --iodepth=32 --bs=128k --direct=1 \
  --ioengine=libaio --max_latency=5s --norandommap \
  --filename=/dev/sdb --name=readjob --rw=read --runtime=24h --time_based

Random Read Random Write

fio --name=global --iodepth=32 --bs=128k --direct=1 \
  --ioengine=libaio \
  --filename=/dev/nvme0n1 \
  --name=randwrite --rw=randwrite \
  --name=randread --rw=randread

fio --name=global --runtime=10m --iodepth=32 --bs=128k --direct=1 \
  --ioengine=libaio \
  --filename=/dev/nvme0n1 \
  --name=randwrite --rw=randwrite \
  --name=randread --rw=randread

fio --name=global --iodepth=32 --bs=128k --direct=1 \
  --ioengine=libaio --max_latency=5s --norandommap \
  --filename=/dev/md0 \
  --name=randwrite --rw=randwrite --runtime=10m --time_based \
  --name=randread --rw=randread --runtime=10m --time_based

Full Sequential Read Write

fio --name=global --iodepth=32 --bs=128k --direct=1 \
  --ioengine=libaio \
  --filename=/dev/nvme0n1 \
  --name=write --rw=write \
  --name=read --rw=read

fio --name=global --runtime=10m --iodepth=32 --bs=128k --direct=1 \
  --ioengine=libaio \
  --filename=/dev/nvme0n1 \
  --name=write --rw=write \
  --name=read --rw=read

keywords

linux benchmarking performance benchmark

Retrieved from "https://aznot.com/index.php?title=Fio&oldid=6147"