View source for Fio

== fio - Flexible IO tester ==

fio - Flexible IO tester (FIO) benchmarking and workload simulation tool

-

fio - Flexible IO Tester - http://git.kernel.dk/?p=fio.git

-

git.kernel.dk Git - http://git.kernel.dk/?p=fio.git
: description: fio - Flexible IO Tester
: owner: Jens Axboe

-

Jens Axboe - Wikipedia - http://en.wikipedia.org/wiki/Jens_Axboe
:Jens Axboe is a Linux kernel hacker. He is the current Linux kernel maintainer of the block layer and other block devices, along with contributing the CFQ I/O scheduler, Noop scheduler, Deadline scheduler and splice (system call) IO architecture. Jens is also the author of the blktrace utility and kernel parts, which provides a way to trace every block IO activity in the Linux kernel. blktrace exists in 2.6.17 and later Linux kernels.
: To facilitate his block layer work in the Linux kernel Jens created the Flexible IO tester (FIO) benchmarking and workload simulation tool. FIO is able to simulate various types of I/O loads, such as synchronous, asynchronous, mmap, etc., as well as specifying the number of threads or processes, read vs. write mix, and various other parameters. FIO was used to set the world record in April 2009 for the highest number of I/Os-per-second (IOPS) in a single system

== Disambiguation ==

See also: [[Fusion-io]] (FIO)

== Source ==

 git clone git://git.kernel.dk/fio.git

or

 git clone https://github.com/axboe/fio.git

== Usage ==

 # 12 hour write test
 fio --name=write_job --filename=/dev/nvme0n1 --iodepth=128 --bs=256k --rw=write --direct=1 --runtime=12h --time_based --ioengine=libaio

 # 5 minute write test
 fio --name=write_job --filename=/dev/nvme0n1 --iodepth=128 --bs=256k --rw=write --direct=1 --runtime=5m --time_based --ioengine=libaio

 # 1 minute write test
 fio --name=write_job --filename=/dev/nvme0n1 --iodepth=128 --bs=256k --rw=write --direct=1 --runtime=1m --time_based --ioengine=libaio

== Installation ==

fio - Flexible IO Tester - http://git.kernel.dk/?p=fio.git

=== Source method ===

Source build and install:
<pre>
# RHEL DEPENDENCIES
sudo yum -y install gcc make wget git libaio-devel zlib-devel
# may require # yum -y --enablerepo=rpmforge install git

# UBUNTU DEPENDENCIES
sudo apt-get install gcc make wget git libaio-dev zlib1g-dev

# MAKE SURE LIBAIO IS INCLUDED, OR PERFORMANCE WILL BE TERRIBLE!

# BUILD
mkdir -p ~/.src ; cd ~/.src
git clone git://git.kernel.dk/fio.git
cd fio
# git checkout fio-2.0.9
# git checkout fio-2.16
# git checkout fio-3.0
# git checkout fio-3.12
git checkout fio-3.16
# git checkout fio-3.19  # newer option
./configure
make clean
make

# INSTALL
cp fio /usr/local/bin
# OR this will include the man and fio_generate_plots page as well
sudo make install
</pre>

=== RPMForge method ===

RPMForge fio install:
 yum -y --enablerepo=rpmforge install fio

=== Freshmeat method ===

fio utility - http://freshmeat.net/projects/fio
* Compiling FIO Benchmark Tool (Linux) - SmarterTrack 4.9 - http://kb.fusionio.com/KB/a26/compiling-fio-benchmark-tool-linux.aspx

 yum -y install gcc libaio-devel make wget
 mkdir -p ~/src ; cd ~/src
 wget "http://freshmeat.net/urls/3aa21b8c106cab742bf1f20d60629e3f"
 tar -zvxf fio-1.57.tar.gz
 cd fio-1.57
 make

 cp fio /usr/local/bin
 # OR
 make install

=== Windows Installation ===

Windows fio - flexible io tester - http://bluestop.org/fio/

* 64bit installer - http://bluestop.org/files/fio/releases/fio-2.6-x64.msi
* 64bit zip - http://bluestop.org/files/fio/releases/fio-2.6-x64.zip
* 32bit installer - http://bluestop.org/files/fio/releases/fio-2.6-x86.msi
* 32bit zip - http://bluestop.org/files/fio/releases/fio-2.6-x86.zip

"This site contains Windows binaries for fio. The main fio site, which conatins the latest code, is at http://git.kernel.dk/?p=fio.git."

: FIO sources which were used to build the current installer can be found in fio.zip.
: The Windows version of FIO uses Cygwin.
: Sources for the Cygwin binaries which are used in the current installer can be found in cygwin_src.zip. 
: http://bluestop.org/fio/cygwin_src.zip

Fio is installed to:
 C:\Program Files\fio\fio.exe

Ported to Windows by Bruce Cran <BCran@fusionio.com>.

Execution:
 "C:\Program Files\fio\fio.exe" --name=job1 --filename=\\.\PhysicalDrive1 --size=100m --rw=write --thread

Windows threading sucks!
 fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.

Errors you will see when the drive fills:
<pre>
fio: io_u error on file \\.\PhysicalDrive1: Input/output error: read offset=1073737728, buflen=4096
fio: pid=7980, err=5/file:io_u.c:1603, func=io_u error, error=Input/output error

job1: (groupid=0, jobs=1): err= 5 (file:io_u.c:1603, func=io_u error, error=Input/output error): pid=7980: Wed Apr 6 20:
</pre>

=== VMware ESXi ===

Support for ESXi is minimal.

<pre>
commit c89318761586dbd77b8f31ce0ed68ff4fc086c89
Author: Jens Axboe <axboe@fb.com>
Date:   Wed Jun 18 15:30:09 2014 -0700

    Add support for compiling for ESX

    With contributions from Ryan Haynes <rhaynes@fusionio.com>

    Signed-off-by: Jens Axboe <axboe@fb.com>
</pre>

From within a <b>ESXi ddk build environment</b>: (will need a separate build for 5.x and 6.x)
 # sinter esxi5
 mkdir -p ~/.src ; cd ~/.src
 git clone git://git.kernel.dk/fio.git
 cd fio
 git checkout c89318761586dbd77b8f31ce0ed68ff4fc086c89

 # build
 make clean
 ./configure --esx
 make

 # verify linked
 ldd fio

 # copy to test server
 scp fio root@vmware-server:/scratch/
 ssh root@vmware-server

 # test
 cd /scratch
 ./fio --help

 # debug
 cd /scratch
 strace ./fio

 # quick write test
 ./fio --name=job1 --filename=/vmfs/volumes/some_datastore/test.txt --size=1m --rw=write
 # quick read test
 ./fio --name=job1 --filename=/vmfs/volumes/some_datastore/test.txt --size=1m

GitHub Issue:
* libaio library not found on ESXi · Issue #80 · axboe/fio · GitHub - https://github.com/axboe/fio/issues/80
** "Sounds like the binary should be statically linked against lbaio instead of dynamically in the case of esxi."
* Disable libaio for ESXi build - bug#80 by kennethburgener · Pull Request #81 · axboe/fio - https://github.com/axboe/fio/pull/81
* Disable libaio for ESXi build - bug#80 by ezrapedersen · Pull Request #138 · axboe/fio - https://github.com/axboe/fio/pull/138

Note: A ESXi 6.0/5.5 build will not work on ESXi 5.0/5.1, but the reverse (5.0 on 6.0) does appear to work.
<pre>
./fio: /lib64/libc.so.6: version `GLIBC_2.7' not found (required by ./fio)
./fio: /lib64/libc.so.6: version `GLIBC_2.10' not found (required by ./fio)
./fio: /lib64/libc.so.6: version `GLIBC_2.6' not found (required by ./fio)
</pre>

== Fusion-io Compiling the fio Utility ==

The fio benchmarking utility is used to verify Linux system performance with an ioDrive. To compile the fio utility, perform the following steps

Grab the latest version of fio source from the tarball link in http://freshmeat.net/projects/fio

Install the necessary standard dependencies. For example:

 $ yum -y install gcc

Install the necessary libaio development headers. For example:

 $ yum -y install libaio-devel

Explode the fio tarball:

 $ tar xjvf fio-X.Y.Z.tar.bz2   
 $ cd fio-X.Y.Z

Build fio for your system:

 $ make

When the rebuild completes successfully, a fio binary is placed in the fio directory.

== Job Options ==

Job Options:
 name=str	# job name (if 'global' apply to all jobs)
 filename=str	# target file or drive (/dev/sdb)
 readwrite=str	# (rw=str) traffic type: read, write, randread, randwrite, rw/readwrite, randrw
 size=int	# size for job (1k, 1m, 1g, 20%, etc)
 blocksize=int	# block size: 4k (default), 1k, 1m, 1g, etc
 ioengine=str	# io engine: sync (default), libaio (linux native asyncrhonous io), etc
 iodepth=int	# I/O queue size
 direct=bool	# use non-buffered I/O (default: 0 - false)
 thinktime=int	# stall job microseconds between issuing I/Os
 nice=int	# run job at this nice value
 cpuload=int	# limit percenage of CPU cycles
 runtime=int	# number of seconds to run
 loops=int	# interations for job (default: 1)
 numjobs=int	# number of clones of job (default: 1)

== Basic Traffic Tests ==

Simple read job:
 fio --name=job1 --filename=/dev/sdb
 fio --name=job1 --filename=/dev/sdb --size=10g

Simple write job:
 fio --name=job1 --filename=/dev/sdb --size=10g --rw=write

Simple read job:
 fio --name=job1 --filename=/dev/sdb --size=10g --rw=read

Simple two process write job:
 fio --name=global --filename=/dev/sdb --size=10g --rw=write --name=job1 --name=job2

Complex Write random traffic to /dev/sdb:
 fio --name=random-writers --ioengine=libaio --iodepth=1 --rw=randwrite --bs=32k --direct=0 --numjobs=1 --loops=2 --filename=/dev/sdb

High bandwidth job:
 fio --name=job1 --ioengine=libaio --iodepth=4 --rw=write --bs=1m --direct=1 --numjobs=1 --size=30g --filename=/dev/sdb

Unaligned DMA: (FreeBSD)
 fio --name=job1 --filename=/dev/ad1 --size=5g --iomem_align=1 --rw=write --bs=256k --thread

Random read job and write job:
 fio --name=global --ioengine=libaio --iodepth=4 --bs=4k --direct=0 --loops=2 --filename=/dev/sdb --name=randread --rw=randread --name=randwrite --rw=randwrite

== Basic Report ==

Read Report:
<pre>
# fio --filename=/dev/sdb --name=job1 --size=10g
job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.11-46-g5a90
Starting 1 process
Jobs: 1 (f=1): [R] [100.0% done] [623.4M/0K/0K /s] [160K/0 /0  iops] [eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=1895: Sat Dec  8 23:20:13 2012
  read : io=10240MB, bw=637471KB/s, iops=159367 , runt= 16449msec
    clat (usec): min=0 , max=2083 , avg= 5.78, stdev=47.34
     lat (usec): min=0 , max=2083 , avg= 5.86, stdev=47.35
    clat percentiles (usec):
     |  1.00th=[    0],  5.00th=[    0], 10.00th=[    1], 20.00th=[    1],
     | 30.00th=[    1], 40.00th=[    1], 50.00th=[    1], 60.00th=[    1],
     | 70.00th=[    1], 80.00th=[    1], 90.00th=[    1], 95.00th=[    1],
     | 99.00th=[  106], 99.50th=[  532], 99.90th=[  564], 99.95th=[  604],
     | 99.99th=[  812]
    bw (KB/s)  : min=597504, max=655360, per=100.00%, avg=637531.25, stdev=10370.41
    lat (usec) : 2=96.31%, 4=2.01%, 10=0.01%, 20=0.09%, 50=0.02%
    lat (usec) : 100=0.52%, 250=0.32%, 500=0.15%, 750=0.55%, 1000=0.04%
    lat (msec) : 2=0.01%, 4=0.01%
  cpu          : usr=8.70%, sys=40.61%, ctx=22011, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=2621440/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=10240MB, aggrb=637470KB/s, minb=637470KB/s, maxb=637470KB/s, mint=16449msec, maxt=16449msec

Disk stats (read/write):
  sdb: ios=42796/0, merge=2575677/0, ticks=26606/0, in_queue=26577, util=81.95%
</pre>

Write Report:
<pre>
# fio --filename=/dev/sdb --name=job1 --size=10g --rw=write
job1: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.11-46-g5a90
Starting 1 process
Jobs: 1 (f=1): [W] [100.0% done] [0K/163.5M/0K /s] [0 /41.9K/0  iops] [eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=1902: Sat Dec  8 23:22:21 2012
  write: io=10240MB, bw=304456KB/s, iops=76113 , runt= 34441msec
    clat (usec): min=1 , max=926829 , avg=12.00, stdev=815.23
     lat (usec): min=2 , max=926829 , avg=12.19, stdev=815.23
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    2], 20.00th=[    2],
     | 30.00th=[    2], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    6], 95.00th=[   10],
     | 99.00th=[   18], 99.50th=[   32], 99.90th=[ 1736], 99.95th=[ 6048],
     | 99.99th=[14656]
    bw (KB/s)  : min=  132, max=741784, per=100.00%, avg=312652.71, stdev=74487.53
    lat (usec) : 2=0.01%, 4=59.53%, 10=35.16%, 20=4.46%, 50=0.54%
    lat (usec) : 100=0.04%, 250=0.11%, 500=0.05%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.04%, 10=0.03%, 20=0.02%, 50=0.01%
    lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  cpu          : usr=11.07%, sys=30.73%, ctx=7795, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=2621440/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=10240MB, aggrb=304455KB/s, minb=304455KB/s, maxb=304455KB/s, mint=34441msec, maxt=34441msec

Disk stats (read/write):
  sdb: ios=83/19960, merge=0/2541727, ticks=5/4637755, in_queue=4655340, util=99.35%
</pre>

== Benchmark ==

simple job file: defines two processes, each randomly reading from a 128MB file:

<pre>
[global]
rw=randread
size=128m

[job1]

[job2]
</pre>

command line version:
 fio --name=global --rw=randread --size=128m --name=job1 --name=job2

---

4 processes each randomly writing to their own files:
* We want to use async io here
* depth of 4 for each file
* increased the buffer size used to 32KB
* numjobs to 4 to fork 4 identical jobs
* 64MB file

<pre>
[random-writers]
ioengine=libaio
iodepth=4
rw=randwrite
bs=32k
direct=0
size=64m
numjobs=4
</pre>

command line version:
 fio --name=random-writers --ioengine=libaio --iodepth=4 --rw=randwrite --bs=32k --direct=0 --size=64m --numjobs=4

---

Fusion-io: fio random 512 read peak IOPS:  (Run for 150 seconds (120 + 30) or 2.5 minutes)
<pre>
[global]
readwrite=randrw
rwmixread=100
blocksize=512
ioengine=libaio
numjobs=4
thread=1
direct=1
iodepth=32
iodepth_batch=16
iodepth_batch_complete=16
group_reporting=1
ramp_time=30
norandommap=1
description=fio random 512 read peak IOPS
time_based=1
runtime=120

[/dev/sdb]
filename=/dev/sdb
</pre>

Command: fio --minimal --output=fio.output fio-job.ini

== Inspecting disk IO performance with fio ==

Inspecting disk IO performance with fio | Linux.com - https://www.linux.com/learn/tutorials/442451-inspecting-disk-io-performance-with-fio
* [http://www.linux.com/feature/131063 Linux.com :: Inspecting disk IO performance with fio] (dead link)
* Linux Disk Benchmarking – IO Performance With fio Tool - http://www.cyberciti.biz/tips/linux-disk-benchmarking-io.html

Linux.com has published article about a new tool called fio:
: fio was created to allow benchmarking specific disk IO workloads. It can issue its IO requests using one of many synchronous and asynchronous IO APIs, and can also use various APIs which allow many IO requests to be issued with a single API call. You can also tune how large the files fio uses are, at what offsets in those files IO is to happen at, how much delay if any there is between issuing IO requests, and what if any filesystem sync calls are issued between each IO request. A sync call tells the operating system to make sure that any information that is cached in memory has been saved to disk and can thus introduce a significant delay. The options to fio allow you to issue very precisely defined IO patterns and see how long it takes your disk subsystem to complete these tasks.

=== random-read-test.fio ===

"The first test you might like to perform is for random read IO performance. This is one of the nastiest IO loads that can be issued to a disk, because it causes the disk head to seek a lot, and disk head seeks are extremely slow operations relative to other hard disk operations. One area where random disk seeks can be issued in real applications is during application startup, when files are requested from all over the hard disk. You specify fio benchmarks using configuration files with an ini file format. You need only a few parameters to get started. rw=randread tells fio to use a random reading access pattern, size=128m specifies that it should transfer a total of 128 megabytes of data before calling the test complete, and the directory parameter explicitly tells fio what filesystem to use for the IO benchmark. On my test machine, the /tmp filesystem is an ext3 filesystem stored on a RAID-5 array consisting of three 500GB Samsung SATA disks. If you don't specify directory, fio uses the current directory that the shell is in, which might not be what you want. "

random-read-test.fio:
<pre>
; random read of 128mb of data

[random-read]
rw=randread
size=128m
directory=/tmp/fio-testing/data
</pre>

Test execution:
<pre>
$ fio random-read-test.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
Starting 1 process
random-read: Laying out IO file(s) (1 file(s) / 128MiB)
Jobs: 1 (f=1): [r] [100.0% done] [  3588/     0 kb/s] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=30598
  read : io=128MiB, bw=864KiB/s, iops=211, runt=155282msec
    clat (usec): min=139, max=148K, avg=4736.28, stdev=6001.02
    bw (KiB/s) : min=  227, max= 5275, per=100.12%, avg=865.00, stdev=362.99
  cpu          : usr=0.07%, sys=1.27%, ctx=32783, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     issued r/w: total=32768/0, short=0/0
     lat (usec): 250=34.92%, 500=0.36%, 750=0.02%, 1000=0.05%
     lat (msec): 2=0.41%, 4=12.80%, 10=44.96%, 20=5.16%, 50=0.94%
     lat (msec): 100=0.37%, 250=0.01%

Run status group 0 (all jobs):
   READ: io=128MiB, aggrb=864KiB/s, minb=864KiB/s, maxb=864KiB/s, mint=155282msec, maxt=155282msec

Disk stats (read/write):
  dm-6: ios=32768/148, merge=0/0, ticks=154728/12490, in_queue=167218, util=99.59%
</pre>


=== random-read-test-aio.fio ===

random-read-test-aio.fio:
<pre>
; random read of 128mb of data with libaio

[random-read]
rw=randread
size=128m
directory=/tmp/fio-testing/data

ioengine=libaio
iodepth=8
direct=1
invalidate=1
</pre>

Execution:
<pre>
$ fio random-read-test-aio.fio
random-read: (groupid=0, jobs=1): err= 0: pid=31318
  read : io=128MiB, bw=2,352KiB/s, iops=574, runt= 57061msec
    slat (usec): min=8, max=260, avg=25.90, stdev=23.23
    clat (usec): min=1, max=124K, avg=13901.91, stdev=12193.87
    bw (KiB/s) : min=    0, max= 5603, per=97.59%, avg=2295.43, stdev=590.60
...
  IO depths    : 1=0.1%, 2=0.1%, 4=4.0%, 8=96.0%, 16=0.0%, 32=0.0%, >=64=0.0%
...
Run status group 0 (all jobs):
   READ: io=128MiB, aggrb=2,352KiB/s, minb=2,352KiB/s, maxb=2,352KiB/s, mint=57061msec, maxt=57061msec
</pre>

== high load test ==

<pre>
fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
  --loops=10000 --size=9654901800 --runtime=600 --rwmixwrite=100 --do_verify=0 --filename=/dev/fioa --thread

fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
  --loops=10000 --size=9654901800 --runtime=600 --rwmixwrite=100 --do_verify=0 --filename=/dev/fiob --thread
</pre>

Source: Fusion thermal monitoring test

Mix: (adjust rwmixwrite)
<pre>
fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
  --loops=10000 --size=9654901800 --runtime=600 --rwmixwrite=60 --do_verify=0 --filename=/dev/fioa --thread
</pre>

== 10 minute load ==

100% Random Write:

 # WRITE: bw=1381MiB/s (1448MB/s), 21.6MiB/s-21.6MiB/s (22.6MB/s-22.6MB/s), io=809GiB (869GB), run=600020-600186msec
 #
 fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
   --loops=10000 --runtime=600 --rwmixwrite=100 --filename=/dev/fioa --thread --allow_mounted_write=1

 # WRITE: bw=1992MiB/s (2088MB/s), 25.5MiB/s-93.6MiB/s (26.7MB/s-98.2MB/s), io=1167GiB (1253GB), run=600022-600158msec
 #
 fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
   --loops=10000 --runtime=600 --rwmixwrite=100 --filename=/dev/nvme0n1 --thread

80% Random Write / 20% Random Read:

 # READ: bw=291MiB/s (305MB/s), 4611KiB/s-4720KiB/s (4722kB/s-4834kB/s), io=170GiB (183GB), run=600181-600195msec
 # WRITE: bw=1164MiB/s (1221MB/s), 18.2MiB/s-18.2MiB/s (19.0MB/s-19.1MB/s), io=682GiB (733GB), run=600181-600195msec
 #
 fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
   --loops=10000 --runtime=600 --rwmixwrite=80 --filename=/dev/fioa --thread --allow_mounted_write=1

 # READ: bw=446MiB/s (468MB/s), 5900KiB/s-10.2MiB/s (6041kB/s-10.7MB/s), io=261GiB (281GB), run=600014-600163msec
 # WRITE: bw=1785MiB/s (1872MB/s), 23.2MiB/s-40.8MiB/s (24.3MB/s-42.8MB/s), io=1046GiB (1123GB), run=600014-600163msec
 #
 fio --rw=randrw --bs=64k --numjobs=64 --iodepth=64 --direct=1 --sync=0 --ioengine=libaio --name=test \
   --loops=10000 --runtime=600 --rwmixwrite=80 --filename=/dev/nvme0n1 --thread

== New ETA ==

Do not report live ETA:
 fio --eta=never ...

== Trim ==

fio can issue trim command:
 fio --name=test --random_generator=lfsr --ioengine=sg --rw=randtrim --filename=/dev/sg4 --iodepth=8 --iodepth_batch=8 --loops=3

== Minimal Output ==

Fio minimal output field indexes - https://www.andypeace.com/fio_minimal.html

"fio, the flexible IO tester, is a very useful tool for benchmarking IO performance. It has an option to produce minimal output, which is very useful when gathering data for later processing, e.g. graphing. The man page for fio describes the output format, but does not number the fields. This means that when extracting fields, one must count the fields in the man page to find the correct index to extract."

<pre>
Field	Description
1	terse version
2	fio version
3	jobname
4	groupid
5	error
Read status:
6	Total I/O (KB)
7	bandwidth (KB/s)
8	IOPS
9	runtime (ms)
Submission latency:
10	min
11	max
12	mean
13	standard deviation
Completion latency:
14	min
15	max
16	mean
17	standard deviation
Completion latency percentiles (20 fields):
18-37	Xth percentile=usec
Total latency:
38	min
39	max
40	mean
41	standard deviation
Bandwidth:
42	min
43	max
44	aggregate percentage of total
45	mean
46	standard deviation
Write status:
47	Total I/O (KB)
48	bandwidth (KB/s)
49	IOPS
50	runtime (ms)
Submission latency:
51	min
52	max
53	mean
54	standard deviation
Completion latency:
55	min
56	max
57	mean
58	standard deviation
Completion latency percentiles (20 fields):
59-78	Xth percentile=usec
Total latency:
79	min
80	max
81	mean
82	standard deviation
Bandwidth:
83	min
84	max
85	aggregate percentage of total
86	mean
87	standard deviation
CPU usage:
88	user
89	system
90	context switches
91	major page faults
92	minor page faults
IO depth distribution:
93	<=1
94	2
95	4
96	8
97	16
98	32
99	>=64
IO latency distribution (microseconds):
100	<=2
101	4
102	10
103	20
104	50
105	100
106	250
107	500
108	750
109	1000
IO latency distribution (milliseconds):
110	<=2
111	4
112	10
113	20
114	50
115	100
116	250
117	500
118	750
118	1000
120	2000
121	>=2000
Disk utilization (1 for each disk used, for disk n, n is zero-based):
122+10n	name
123+10n	read ios
124+10n	write ios
125+10n	read merges
126+10n	write merges
127+10n	read ticks
128+10n	write ticks
129+10n	read in-queue time
130+10n	write in-queue time
131+10n	disk utilization
132+10n	percentage
Error Info (dependent on continue_on_error, default off):
F-1	total # errors
F-0	first error code
newline	text description (if provided in config - appears on newline)
</pre>

== Verification Testing ==

 verify=str
:If writing to a file, fio can verify the file contents after each iteration of the job. Each verification method also implies verification of special header, which is written to the beginning of each block. This header also includes meta information, like offset of the block, block number, timestamp when block was written, etc. verify can be combined with verify_pattern option. 

See https://fio.readthedocs.io/en/latest/fio_doc.html#verification

== Samples ==

Running a read zero job: (Kep K.)
<pre>
[global]
readwrite=randrw
rwmixread=100
blocksize=512
ioengine=libaio
numjobs=1
thread=0
direct=1
iodepth=32
iodepth_batch=16
iodepth_batch_complete=16
group_reporting=1
#ramp_time=5
norandommap=1
description=fio random 512 read zero IOPS
time_based=1
runtime=30
randrepeat=0


[/dev/fioa]
filename=/dev/fioa
cpus_allowed=1
startdelay=1

[/dev/fioa]
filename=/dev/fioa
cpus_allowed=2
startdelay=2

[/dev/fioa]
filename=/dev/fioa
cpus_allowed=3
startdelay=3

[/dev/fioa]
filename=/dev/fioa
cpus_allowed=4
startdelay=4
</pre>

=== Linked Clones Example ===

by Kenneth Burgener

<pre>
[random-writers]
ioengine=libaio
iodepth=1
rw=randwrite
bs=32k
direct=0
; nice=19
; thinktime=10000
; size=1g
numjobs=1
loops=99999
; cpuload=10
filename=/dev/sdb1
</pre>

Command line version:
 fio --name=random-writers --ioengine=libaio --iodepth=1 --rw=randwrite --bs=32k --direct=0 --numjobs=1 --loops=99999 --filename=/dev/sdb

=== Bandwidth Test ===

<pre>
[global]
readwrite=randrw
rwmixread=0
blocksize=1M
ioengine=libaio
numjobs=4
thread=0
direct=1
iodepth=32
iodepth_batch=16
iodepth_batch_complete=16
group_reporting=1
ramp_time=5
norandommap=1
description=fio random 1M write peak BW
time_based=1
runtime=60
randrepeat=0

[/dev/sdb]
filename=/dev/sdb
</pre>

Only changing the blocksize in the following fio config:
<pre>
|8K|  WRITE: io=24378MB, aggrb=416040KB/s, minb=426025KB/s, 
|8K| maxb=426025KB/s, mint=60002msec, maxt=60002msec|
|16K|  WRITE: io=37994MB, aggrb=648328KB/s, minb=663888KB/s, 
|16K| maxb=663888KB/s, mint=60009msec, maxt=60009msec|
|32K|  WRITE: io=50756MB, aggrb=866155KB/s, minb=886943KB/s, 
|32K| maxb=886943KB/s, mint=60006msec, maxt=60006msec|
|64K|  WRITE: io=60554MB, aggrb=1009.5MB/s, minb=1033.3MB/s, 
|64K| maxb=1033.3MB/s, mint=60011msec, maxt=60011msec|
|128K|  WRITE: io=18419MB, aggrb=314246KB/s, minb=321788KB/s, 
|128K| maxb=321788KB/s, mint=60020msec, maxt=60020msec|
|256K|  WRITE: io=21555MB, aggrb=366735KB/s, minb=375536KB/s, 
|256K| maxb=375536KB/s, mint=60186msec, maxt=60186msec|
|512K|  WRITE: io=29526MB, aggrb=502411KB/s, minb=514469KB/s, 
|512K| maxb=514469KB/s, mint=60179msec, maxt=60179msec|
|1M|  WRITE: io=38220MB, aggrb=649344KB/s, minb=664928KB/s, 
|1M| maxb=664928KB/s, mint=60272msec, maxt=60272msec|
|2M|  WRITE: io=34552MB, aggrb=585724KB/s, minb=599781KB/s, 
|2M| maxb=599781KB/s, mint=60406msec, maxt=60406msec|
|4M|  WRITE: io=34992MB, aggrb=588999KB/s, minb=603135KB/s, 
|4M| maxb=603135KB/s, mint=60835msec, maxt=60835msec|
</pre>

Another run involves reducing these:
 iodepth=16 iodepth_batch=8 iodepth_batch_complete=8

<!-- References: https://sup.fusionio.com/cases/browse/CASE-20398 -->

=== Windows Sample ===

Thane H.: Here is a sample ini file that uses raw devices:
<pre>
[global]
rw=read
name=fire
verify=crc32c
bs=4096b
iodepth=256
size=64g
thread
[thread_fct0-0]
filename=\\.\PhysicalDrive1
[thread_fct0-1]
filename=\\.\PhysicalDrive1
[thread_fct2-0]
filename=\\.\PhysicalDrive0
[thread_fct2-1]
filename=\\.\PhysicalDrive0
</pre>

Another sample:
 # fio --name=job1 --filename=e: --size=5g --rw=write --bs=256k --thread

List devices:
 wmic diskdrive list brief

=== Unaligned DMA ===

<pre>
iomem_align=int This indiciates the memory alignment of the IO memory
                buffers. Note that the given alignment is applied to
                the first IO unit buffer, if using iodepth the
                alignment of the following buffers are given by the
                bs used. In other words, if using a bs that is a multiple
                of the page sized in the system, all buffers will be
                aligned to this value. If using a bs that is not page
                aligned, the alignment of subsequent IO memory buffers
                is the sum of the iomem_align and bs used.
</pre>

Linux:
 fio --name=job1 --filename=/dev/fioa --size=5g --iomem_align=1 --rw=write --bs=256k

FreeBSD:
 fio --name=job1 --filename=/dev/ad1 --size=5g --iomem_align=1 --rw=write --bs=256k --thread
 fio --name=job1 --filename=/dev/ad1 --size=1M --iomem_align=1 --direct=1 --rw=write --bs=256k --thread

Windows 7:
 C:\Program Files\fio>fio --name=job1 --filename=e: --size=5g --rw=write --bs=256k --thread --iomem_align=1 --direct=1

Windows 7 Raw Device: (Must run as Administrator)
 C:\Program Files\fio>fio --name=job1 --filename=\\.\PHYSICALDRIVE1 --size=5g --rw=write --bs=256k --thread --iomem_align=1 --direct=1

=== Misc ===

fio-4k-r-qd8.ini
<pre>
[global]
readwrite=read
rwmixread=100
blocksize=4k
ioengine=aio
numjobs=16
thread=1
direct=1
iodepth=16
iodepth_batch=16
iodepth_batch_complete=16
group_reporting=1
ramp_time=5
norandommap=1
description=fio 4k read IOPS QD 8
time_based=1
runtime=300
randrepeat=0

[/dev/nbd0]
filename=/dev/nbd0
</pre>

=== More Examples ===

Linux:
<pre>
fio --name=job1 --filename=/dev/sdb --iodepth=1 --bs=256k --rw=write --direct=1 --runtime=30 --ioengine=libaio

fio --name=job1 --filename=/dev/sdb --iodepth=1 --bs=256k --rw=write --direct=1 --runtime=30

fio --name=job1 --filename=/dev/sdb --iodepth=1 --bs=256k --rw=read --direct=1 --runtime=30

fio --name=job1 --filename=/dev/sdb --iodepth=32 --bs=256k --rw=write --direct=1 --runtime=60

fio --name=job1 --filename=/dev/sdb --iodepth=32 --bs=256k --rw=read --direct=1 --runtime=60

fio --name=job1 --filename=/dev/sdb --iodepth=32 --bs=256k --rw=write --direct=1 --name=job2 --rw=read --filename=/dev/sdb --iodepth=32 --direct=1 --bs=256k

fio --name=job1 --filename=/dev/sdb --iodepth=32 --bs=256k --rw=write --direct=1 --thread --runtime=120

fio --name=job1 --filename=/dev/sdb --iodepth=32 --bs=256k --rw=read --direct=1 --thread --runtime=120

fio --name=global --ioengine=libaio --iodepth=32 --bs=4k --direct=0 --loops=2 --filename=/dev/sdb --name=randread --rw=randread --name=randwrite --rw=randwrite

fio --output=output.txt --filename=/dev/nvme0n1 --iodepth=128 --bs=4k --thread=2 --direct=1 --ioengine=libaio --name=write-phase --rw=write --name=read-phase --rw=read

fio --output=output.txt --filename=/dev/nvme0n1 --iodepth=128 --bs=4k --thread=1 --direct=1 --ioengine=libaio --name=write-phase \
    --do_verify=0 --rw=write --name=read-phase --stonewall --do_verify=0 --time_based=1 --loops=999 --runtime=24h --rw=read
</pre>

Windows:
<pre>
wmic diskdrive list brief

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=1 --bs=256k --rw=write --direct=1 --thread --runtime=30

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=1 --bs=256k --rw=read --direct=1 --thread --runtime=30

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=32 --bs=256k --rw=write --direct=1 --thread

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=32 --bs=256k --rw=read --direct=1 --thread

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=32 --bs=256k --rw=write --direct=1 --name=job2 --rw=read --filename=\\.\PhysicalDrive1 --iodepth=32 --direct=1 --bs=256k --thread

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=32 --bs=256k --rw=write --direct=1 --thread --runtime=120

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=32 --bs=256k --rw=read --direct=1 --thread --runtime=120

fio --name=job1 --filename=\\.\PhysicalDrive1 --iodepth=1 --bs=4k --rw=write --direct=1 --thread --runtime=60

fio --name=global --iodepth=32 --bs=4k --direct=0 --loops=2 --filename=\\.\PhysicalDrive1 --name=randread --rw=randread --name=randwrite --rw=randwrite
</pre>

==== nvme random write ====

 fio --name=rand_write_job --filename=/dev/nvme0n1 --iodepth=128 --bs=4k --rw=randwrite --direct=1 --runtime=24h --time_based --ioengine=libaio

 fio --name=rand_write_job --filename=/dev/nvme0n1 --iodepth=128 --bs=4k --rw=randwrite --direct=1 --runtime=24h --time_based --ioengine=libaio norandommap random_generator=tausworthe


<pre>
norandommap random_generator=tausworthe
</pre>

:"Normally fio will cover every block of the file when doing random I/O. If this parameter is given, a new offset will be chosen without looking at past I/O history. This parameter is mutually exclusive with verify."

==== Output to file statistics ====

 fio --name=global --status-interval=10 --output fio.txt --output-format=terse --eta-newline=10 --numjobs=1 --iodepth=32 --direct=1 --ioengine=libaio --max_latency=5s \
   --norandommap --filename=/dev/sda --name=precondjob --rw=write --bs=128k --fill_device=1

 fio --name=global --status-interval=60 --output fio.txt --eta-newline=10 --numjobs=1 --iodepth=32 --direct=1 --ioengine=libaio --max_latency=5s --norandommap \
    --filename=/dev/sda --name=precondjob --rw=write --bs=128k --size=2GB --stonewall --name=readjob --loops=10 --rw=read --bs=128k --stonewall

Report full status every 1 minute:
 --status-interval=1m

==== fill then read forever ====

Limit to only first 20GB

fill-read-forever.fio:
<pre>
[write]
rw=write
bs=128k
direct=1
ioengine=libaio
iodepth=128
size=20G
filename=/dev/nvme0n1
stonewall

[read]
rw=read
bs=128k
direct=1
ioengine=libaio
iodepth=128
size=20G
filename=/dev/nvme0n1
runtime=9999h
time_based
stonewall
</pre>

== Standard ==

 /dev/sdb
 /dev/nvme0n1


=== Precondition Write (full drive write) ===

 fio --name=global --iodepth=32 --bs=128k --direct=1 --ioengine=libaio \
   --filename=/dev/nvme0n1 --name=precondjob --rw=write --fill_device=1

=== Sequential Write (time based) ===

 fio --name=global --iodepth=32 --bs=128k --direct=1 --ioengine=libaio \
   --filename=/dev/nvme0n1 --name=writejob --rw=write --runtime=24h --time_based

Precondition: (fill drive)
 fio --name=global --iodepth=32 --bs=128k --direct=1 --ioengine=libaio \
   --filename=/dev/nvme0n1 --name=precondition --rw=write --fill_device=1

=== Sequential Read ===

 fio --name=global --iodepth=32 --bs=128k --direct=1 --ioengine=libaio \
   --filename=/dev/nvme0n1 --name=readjob --rw=read --runtime=24h --time_based

=== Sequential Read (full drive read) ===

 fio --name=global --iodepth=32 --bs=128k --direct=1 --ioengine=libaio \
   --filename=/dev/nvme0n1 --name=readjob --rw=read

== Standard with Status Interval ==

=== Precondition Write ===

 fio --name=global --status-interval=60 --output fio-precond.txt --iodepth=32 --bs=128k --direct=1 \
   --ioengine=libaio --max_latency=5s --norandommap \
   --filename=/dev/sdb --name=precondjob --rw=write --fill_device=1

=== Sequential Write ===

 fio --name=global --status-interval=60 --output fio-write.txt --iodepth=32 --bs=128k --direct=1 \
   --ioengine=libaio --max_latency=5s --norandommap \
   --filename=/dev/sdb --name=writejob --rw=write --runtime=24h --time_based

With Precondition:
 fio --name=global --status-interval=60 --output fio-precond.txt --iodepth=128 --bs=128k --direct=1 \
   --ioengine=libaio --max_latency=5s --norandommap \
   --filename=/dev/nvme0n1 \
   --name=precondjob --rw=write --fill_device=1 \
   --name=writejob --rw=write --runtime=24h --time_based --stonewall

=== Sequential Read ===

 fio --name=global --status-interval=60 --output fio-read.txt --iodepth=32 --bs=128k --direct=1 \
   --ioengine=libaio --max_latency=5s --norandommap \
   --filename=/dev/sdb --name=readjob --rw=read --runtime=24h --time_based

== Random Read Random Write ==

 fio --name=global --iodepth=32 --bs=128k --direct=1 \
   --ioengine=libaio \
   --filename=/dev/nvme0n1 \
   --name=randwrite --rw=randwrite \
   --name=randread --rw=randread

 fio --name=global --runtime=10m --iodepth=32 --bs=128k --direct=1 \
   --ioengine=libaio \
   --filename=/dev/nvme0n1 \
   --name=randwrite --rw=randwrite \
   --name=randread --rw=randread

 fio --name=global --iodepth=32 --bs=128k --direct=1 \
   --ioengine=libaio --max_latency=5s --norandommap \
   --filename=/dev/md0 \
   --name=randwrite --rw=randwrite --runtime=10m --time_based \
   --name=randread --rw=randread --runtime=10m --time_based

== Full Sequential Read Write ==

 fio --name=global --iodepth=32 --bs=128k --direct=1 \
   --ioengine=libaio \
   --filename=/dev/nvme0n1 \
   --name=write --rw=write \
   --name=read --rw=read

 fio --name=global --runtime=10m --iodepth=32 --bs=128k --direct=1 \
   --ioengine=libaio \
   --filename=/dev/nvme0n1 \
   --name=write --rw=write \
   --name=read --rw=read

== keywords ==

linux benchmarking performance benchmark

[[Category:Linux]]

[[Category:Network Admin]]

[[Category:Benchmark]]

[[Category:Fusion-io]]