Disk formatting


Disk formatting is the process of preparing a data storage device such as a hard disk drive, solid-state drive, floppy disk or USB flash drive for initial use. In some cases, the formatting operation may also create one or more new file systems. The first part of the formatting process that performs basic medium preparation is often referred to as "low-level formatting". Partitioning is the common term for the second part of the process, making the data storage device visible to an operating system. The third part of the process, usually termed "high-level formatting" most often refers to the process of generating a new file system. In some operating systems all or parts of these three processes can be combined or repeated at different levels and the term "format" is understood to mean an operation in which a new disk medium is fully prepared to store files. Some formatting utilities allow distinguishing between a quick format, which does not erase all existing data and a long option that does erase all existing data.
As a general rule, formatting a disk by default leaves most if not all existing data on the disk medium; some or most of which might be recoverable with privileged or special tools. Special tools can remove user data by a single overwrite of all files and free space.

History

A block, a contiguous number of bytes, is the minimum unit of storage that is read from and written to a disk by a disk driver. The earliest disk drives had fixed block sizes but starting with the 1301 IBM marketed subsystems that featured variable block sizes: a particular track could have blocks of different sizes. The disk subsystems on the IBM System/360 expanded this concept in the form of Count Key Data and later Extended Count Key Data ; however the use of variable block size in HDDs fell out of use in the 1990s; one of the last HDDs to support variable block size was the IBM 3390 Model 9, announced May 1993.
Modern hard disk drives, such as Serial attached SCSI and Serial ATA drives, appear at their interfaces as a contiguous set of fixed-size blocks; for many years 512 bytes long but beginning in 2009 and accelerating through 2011, all major hard disk drive manufacturers began releasing hard disk drive platforms using the Advanced Format of 4096 byte logical blocks.
Floppy disks generally only used fixed block sizes but these sizes were a function of the host's OS and its interaction with its controller so that a particular type of media would have different block sizes depending upon the host OS and controller.
Optical discs generally only use fixed block sizes.

Disk formatting process

Formatting a disk for use by an operating system and its applications typically involves three different processes.
  1. Low-level formatting marks the surfaces of the disks with markers indicating the start of a recording block and other information like block CRC to be used later, in normal operations, by the disk controller to read or write data. This is intended to be the permanent foundation of the disk, and is often completed at the factory.
  2. Partitioning divides a disk into one or more regions, writing data structures to the disk to indicate the beginning and end of the regions. This level of formatting often includes checking for defective tracks or defective sectors.
  3. High-level formatting creates the file system format within a disk partition or a logical volume. This formatting includes the data structures used by the OS to identify the logical drive or partition's contents. This may occur during operating system installation, or when adding a new disk. Disk and distributed file system may specify an optional boot block, and/or various volume and directory information for the operating system.

    Low-level formatting of floppy disks

The low-level format of floppy disks is performed by the disk drive's controller.
For a standard 1.44 MB floppy disk, low-level formatting normally writes 18 sectors of 512 bytes to each of 160 tracks of the floppy disk, providing 1,474,560 bytes of storage on the disk.
Physical sectors are actually larger than 512 bytes, as in addition to the 512 byte data field they include a sector identifier field, CRC bytes and gaps between the fields. These additional bytes are not normally included in the quoted figure for overall storage capacity of the disk.
Different low-level formats can be used on the same media; for example, large records can be used to cut down on inter-record gap size.
Several freeware, shareware and free software programs allowed considerably more control over formatting, allowing the formatting of high-density 3.5" disks with a capacity up to 2 MB.
Techniques used include:
Linux supports a variety of sector sizes, and DOS and Windows support a large-record-size DMF-formatted floppy format.

Low-level formatting (LLF) of hard disks

Hard disk drives prior to the 1990s typically had a separate disk controller that defined how data was encoded on the media. With the media, the drive and/or the controller possibly procured from separate vendors, users were often able to perform low-level formatting. Separate procurement also had the potential of incompatibility between the separate components such that the subsystem would not reliably store data.
User instigated low-level formatting of hard disk drives was common for minicomputer and personal computer systems until the 1990s. IBM and other mainframe system vendors typically supplied their hard disk drives with a low-level format. Typically this involved subdividing each track on the disk into one or more blocks which would contain the user data and associated control information. Different computers used different block sizes and IBM notably used variable block sizes but the popularity of the IBM PC caused the industry to adopt a standard of 512 user data bytes per block by the middle 1980s.
Depending upon the system, low-level formatting was generally done by an operating system utility. IBM compatible PCs used the BIOS, which is invoked using the MS-DOS debug program, to transfer control to a routine hidden at different addresses in different BIOSes.

Transition away from LLF

Starting in the late 1980s, driven by the volume of IBM compatible PCs, HDDs became routinely available pre-formatted with a compatible low-level format. At the same time, the industry moved from historical bit serial interfaces to modern bit serial interfaces and word serial interfaces wherein the low level format was performed at the factory. Accordingly, it is not possible for an end user to low-level format a modern hard disk drive.

Disk reinitialization

While it is generally impossible to perform a complete LLF on most modern hard drives outside the factory, the term "low-level format" is still used for what could be called the reinitialization of a hard drive to its factory configuration.
The present ambiguity in the term low-level format seems to be due to both inconsistent documentation on web sites and the belief by many users that any process below a high-level format must be called a low-level format. Since much of the low level formatting process can today only be performed at the factory, various drive manufacturers describe reinitialization software as LLF utilities on their web sites. Since users generally have no way to determine the difference between a complete LLF and reinitialization, both the misinformed user and mixed signals from various drive manufacturers have perpetuated this error. Note: whatever possible misuse of such terms may exist, many sites do make such reinitialization utilities available, to both overwrite every byte and check for damaged sectors on the hard disk.
Reinitialization should include identifying any sectors which cannot be written to and read back from the drive, correctly. The term has, however, been used by some to refer to only a portion of that process, in which every sector of the drive is written to; usually by writing a specific value to every addressable location on the disk.
Traditionally, the physical sectors were initialized with a fill value of 0xF6 as per the INT 1Eh's Disk Parameter Table during format on IBM compatible machines. This value is also used on the Atari Portfolio. 8-inch CP/M floppies typically came pre-formatted with a value of 0xE5, and by way of Digital Research this value was also used on Atari ST and some Amstrad formatted floppies. Amstrad otherwise used 0xF4 as a fill value. Some modern formatters wipe hard disks with a value of 0x00 instead, sometimes also called zero-filling, whereas a value of 0xFF is used on flash disks to reduce wear. The latter value is typically also the default value used on ROM disks. Some advanced formatting tools allow configuring the fill value.
One popular method for performing only the zero-fill operation on a hard disk is by writing zero-value bytes to the drive using the Unix dd utility with the /dev/zero stream as the input file and the drive itself as the output file. This command may take many hours to complete, and can erase all files and file systems.
Another method for SCSI disks may be to use the sg_format command to issue a low-level SCSI Format Unit Command.
Zero-filling a drive is not necessarily a secure method of erasing sensitive data, or of preparing a drive for use with an encrypted filesystem. Zero-filling voids the plausible deniability of the process.

Partitioning

Partitioning is the process of writing information into blocks of a storage device or medium that allows access by an operating system. Some operating systems allow the device to appear as multiple devices; i.e. partitioned into multiple devices.
On MS-DOS, Microsoft Windows, and UNIX-based operating systems this is normally done with a partition editor, such as fdisk, GNU Parted, or Disk Utility. These operating systems support multiple partitions.
In current IBM mainframe OSs derived from OS/360 and DOS/360, such as z/OS and z/VSE, this is done by the INIT command of the ICKDSF utility. These OSs support only a single partition per device, called a volume. The ICKDSF functions include creating a volume label and writing a Record 0 on every track.
Floppy disks are not partitioned; however depending upon the OS they may require volume information in order to be accessed by the OS.
Partition editors and ICKDSF today do not handle low level functions for HDDs and optical disc drives such as writing timing marks, and they cannot reinitialize a modern disk that has been degaussed or otherwise lost the factory formatting.

High-level formatting

High-level formatting is the process of setting up an empty file system on a disk partition or logical volume and, for PCs, installing a boot sector. This is a fast operation, and is sometimes referred to as quick formatting.
The entire logical drive or partition may optionally be scanned for defects, which may take considerable time.
In the case of floppy disks, both high- and low-level formatting are customarily performed in one pass by the disk formatting software. 8-inch floppies typically came low-level formatted and were filled with a format filler value of 0xE5. Since the 1990s, most 5.25-inch and 3.5-inch floppies have been shipped pre-formatted from the factory as DOS FAT12 floppies.
In current IBM mainframe operating systems derived from OS/360 or DOS/360, this may be done as part of allocating a file, by a utility specific to the file system or, in some older access methods, on the fly as new data are written.

Host protected area

The host protected area, sometimes referred to as hidden protected area, is an area of a hard drive that is high level formatted such that the area is not normally visible to its operating system.

Reformatting

Reformatting is a [|high-level formatting] performed on a functioning disk drive to free the medium of its contents. Reformatting is unique to each operating system because what actually is done to existing data varies by OS. The most important aspect of the process is that it frees disk space for use by other data. To actually "erase" everything requires overwriting each block of data on the medium; something that is not done by many high-level formatting utilities.
Reformatting often carries the implication that the operating system and all other software will be reinstalled after the format is complete. Rather than fixing an installation suffering from malfunction or security compromise, it may be necessary to simply reformat everything and start from scratch. Various colloquialisms exist for this process, such as "wipe and reload", "nuke and pave", "reimage", etc.

Formatting

DOS, OS/2 and Windows

format command: Under MS-DOS, PC DOS, OS/2 and Microsoft Windows, disk formatting can be performed by the format command. The format program usually asks for confirmation beforehand to prevent accidental removal of data, but some versions of DOS have an undocumented /AUTOTEST option; if used, the usual confirmation is skipped and the format begins right away. The WM/FormatC macro virus uses this command to format drive C: as soon as a document is opened.
Unconditional format: There is also the /U parameter that performs an unconditional format which under most circumstances overwrites the entire partition, preventing the recovery of data through software. Note however that the /U switch only works reliably with floppy diskettes. Technically because unless /Q is used, floppies are always low level formatted in addition to high-level formatted. Under certain circumstances with hard drive partitions, however, the /U switch merely prevents the creation of unformat information in the partition to be formatted while otherwise leaving the partition's contents entirely intact. In such cases, the user's data remain ripe for recovery with specialist tools such as EnCase or disk editors. Reliance upon /U for secure overwriting of hard drive partitions is therefore inadvisable, and purpose-built tools such as DBAN should be considered instead.
Overwriting: In Windows Vista and upwards the non-quick format will overwrite as it goes. Not the case in Windows XP and below.
OS/2: Under OS/2, if you use the /L parameter, which specifies a long format, then format will overwrite the entire partition or logical drive. Doing so enhances the ability of CHKDSK to recover files.

Unix-like operating systems

High-level formatting of disks on these systems is traditionally done using the mkfs command. On Linux mkfs is typically a wrapper around filesystem-specific commands which have the name mkfs.fsname, where fsname is the name of the filesystem with which to format the disk. Some filesystems which are not supported by certain implementations of mkfs have their own manipulation tools; for example Ntfsprogs provides a format utility for the NTFS filesystem.
Some Unix and Unix-like operating systems have higher-level formatting tools, usually for the purpose of making disk formatting easier and/or allowing the user to partition the disk with the same tool. Examples include GNU Parted and the Disk Utility application on Mac OS X.

Recovery of data from a formatted disk

As in file deletion by the operating system, data on a disk are not fully erased during every high-level format. Instead, the area on the disk containing the data is merely marked as available, and retains the old data until it is overwritten. If the disk is formatted with a different file system than the one which previously existed on the partition, some data may be overwritten that wouldn't be if the same file system had been used. However, under some file systems, the file indexes may not be written to the same exact locations. And if the partition size is increased, even FAT file systems will overwrite more data at the beginning of that new partition.
From the perspective of preventing the recovery of sensitive data through recovery tools, the data must either be completely overwritten with random data before the format, or the format program itself must perform this overwriting, as the DOS FORMAT command did with floppy diskettes, filling every data sector with the format filler byte value.
However, there are applications and tools, especially used in forensic information technology, that can recover data that has been conventionally erased. In order to avoid the recovery of sensitive data, governmental organization or big companies use information destruction methods like the Gutmann method. For average users there are also special applications that can perform complete data destruction by overwriting previous information. Although there are applications that perform multiple writes to assure data erasure, any single write over old data is generally all that is needed on modern hard disk drives. The ATA Secure Erase can be performed by disk utilities to quickly and thoroughly wipe drives. Degaussing is another option; however, this may render the drive unusable.