Sparse File

File System Support

"Unixy" filesystems : ext2, ext3, ufs, xfs, reiserfs, jfs,... all support Sparse files

FAT16 and FAT32 do not

NTFS does support Sparse files

References:

• Sparse files - MLDonkey - http://mldonkey.sourceforge.net/Sparse_files#FAT16.2C_FAT32
• File System Functionality Comparison (Windows) - https://msdn.microsoft.com/en-us/library/windows/desktop/ee681827%28v=vs.85%29.aspx

Sparse File

Wiki: Sparse Files

In computer science, a sparse file is a type of computer file that attempts to use file system space more efficiently. When space has been allocated to a file but not actually filled with data it is not written to the file system. Instead, meta-information about these "empty" regions is stored until they are filled with data.

The obvious advantage of sparse files is that storage is only allocated when actually needed. Large files can be created even if there isn't enough free space yet. A disadvantage is that sparse files can become very fragmented. Also, filling up partitions to the maximum can have unpleasant effects.

Will create a file of one megabyte in size, but with only one byte actually stored on disk:

dd if=/dev/zero of=sparse-file bs=1 count=1 seek=1M

Sparse Tools

View sparse size (in blocks)

# both will report the full file size, but the block size will show sparse size
ls -ls [file]
stat [file]

du -s -B1 --apparent-size sparse-file
du -s -B1 sparse-file

tar

-S, --sparse	# handle sparse files efficiently

cp - # control creation of sparse files

# cp tries to detect sparse files automatically.  This control can be forced
cp --sparse=WHEN [...]
# --sparse=auto
# --sparse=always
# --sparse=never

man cp:
  By  default,  sparse SOURCE files are detected by a crude heuristic and
  the corresponding DEST file is made sparse as well.  That is the behav-
  ior  selected  by  --sparse=auto.   Specify --sparse=always to create a
  sparse DEST file whenever  the  SOURCE  file  contains  a  long  enough
  sequence  of  zero  bytes.   Use  --sparse=never to inhibit creation of
  sparse files.

This is especially useful if a sparse-file has somehow become non-sparse (i.e. the empty blocks have been written out to disk in full). Disk space can be recovered by doing:

cp --sparse=always formerly-sparse-file recovered-sparse-file

rsync

-S, --sparse	# handle sparse files efficiently
rsync -aP --sparse local-file remote-host:remote-file

cpio

--sparse

Not supported tools:

# Using such a utility would make a sparse file balloon in size.
pax, scp, sftp, and ftp

keywords

sparse file