Dd (Unix)
dd is a command-line utility for Unix and Unix-like operating systems, the primary purpose of which is to convert and copy files.
On Unix, device drivers for hardware and special device files appear in the file system just like normal files; can also read and/or write from/to these files, provided that function is implemented in their respective driver. As a result, can be used for tasks such as backing up the boot sector of a hard drive, and obtaining a fixed amount of random data. The program can also perform conversions on the data as it is copied, including byte order swapping and conversion to and from the ASCII and EBCDIC text encodings.
History
The name is an allusion to the DD statement found in IBM's Job Control Language, in which the initials stand for "Data Definition". The command's syntax resembles a JCL statement more than other Unix commands do, so much that Eric S. Raymond says "the interface design was clearly a prank". The interface is redesigned in Plan 9's dd command to use a command-line option style.Originally intended to convert between ASCII and EBCDIC, first appeared in Version 5 Unix. The command is specified since the X/Open Portability Guide issue 2 of 1987. This is inherted into IEEE Std 1003.1-2008, which is part of the Single UNIX Specification.
The version of
dd bundled in GNU coreutils was written by Paul Rubin, David MacKenzie, and Stuart Kemp.Usage
The command line syntax of differs from many other Unix programs. It uses the syntax for its command-line options rather than the more standard or formats. By default, reads from stdin and writes to stdout, but these can be changed by using the and options.Certain features of will depend on the computer system capabilities, such as 's ability to implement an option for direct memory access. Sending a SIGINFO signal to a running process makes it print I/O statistics to standard error once and then continue copying. can read standard input from the keyboard. When end-of-file is reached, will exit. Signals and EOF are determined by the software. For example, Unix tools ported to Windows vary as to the EOF: Cygwin uses and MKS Toolkit uses .
The non-standardized parts of dd invocation vary among implementations.
Output messages
On completion, prints to the stderr stream about statistics of the data transfer. The format is standardized in POSIX. The manual page for GNU dd does not describe this format, but the BSD manuals do.Each of the "Records in" and "Records out" lines shows the number of complete blocks transferred + the number of partial blocks, e.g. because the physical medium ended before a complete block was read, or a physical error prevented reading the complete block.
Block size
A block is a unit measuring the number of bytes that are read, written, or converted at one time. Command-line options can specify a different block size for input/reading compared to output/writing, though the block size option will override both and. The default value for both input and output block sizes is 512 bytes. The option for copying is measured in blocks, as are both the count for reading and count for writing. Conversion operations are also affected by the "conversion block size".The value provided for block size options is interpreted as a decimal integer number of bytes. It can also contain suffixes to indicate that the block size is an integer number of larger units than bytes. POSIX only specifies the suffixes for 512 and for 1024. Implementation differ on the additional suffixes they support: BSD uses lowercase , , and so on for tebibytes, exbibytes, pebibytes, zebibytes, and yobibytes, while GNU uses and for the same units, with,, and used for their SI unit counterparts. For example, for GNU, indicates a blocksize of 16 mebibytes and specifies 3000 bytes.
Additionally, some implementations understand the character as a multiplication operator for both block size and count parameters. For example, is interpreted as 2 × 80 × 18 × 512 =, the exact size of a 1440 KiB floppy disk. This is required in POSIX, but GNU does not seem to support it. As a result, it is more portable to use the POSIX shell arithmetic syntax of
Block size has an effect on performance of copying commands. Doing many small reads or writes is often slower than doing fewer large ones. Using large blocks requires more RAM and can complicate error recovery. When is used with variable-block-size devices such as tape drives or networks, the block size may determine the tape record size or packet size, depending on the network protocol used.
Uses
The command can be used for a variety of purposes. For plain-copying commands it tends to be slower than the domain-specific alternatives, but it excels at its unique ability to "overwrite or truncate a file at any point or seek in a file", a fairly low-level interface to the Unix file API.The examples below assume the use of GNU dd, mainly in the block size argument. To make them portable, replace e.g. with the shell arithmetic expression or .
Data transfer
can duplicate data across files, devices, partitions and volumes. The data may be input or output to and from any of these; but there are important differences concerning the output when going to a partition. Also, during the transfer, the data can be modified using the options to suit the medium.An attempt to copy the entire disk using may omit the final block if it is of an unexpected length; whereas may succeed. The source and destination disks should have the same size.
| Creates an ISO disk image from a CD-ROM, DVD or Blu-ray disk. | |
| Restores a hard disk drive from a previously created image. | |
| Create an image of the partition sdb2, using a 64 MiB block size. | |
| Clones one partition to another. | |
| Clones a hard disk drive "ad0" to "ad1". |
The option means to keep going if there is an error, while the option causes output blocks to be padded.
In-place modification
can modify data in place. For example, this overwrites the first 512 bytes of a file with null bytes:The conversion option means do not truncate the output file — that is, if the output file already exists, just replace the specified bytes and leave the rest of the output file alone. Without this option, would create an output file 512 bytes long.
Master boot record backup and restore
The example [|above] can also be used to back up and restore any region of a device to a file, such as a master boot record.To duplicate the first two sectors of a floppy disk:
Disk wipe
For security reasons, it is sometimes necessary to have a disk wipe of a discarded device. This can be achieved by a "data transfer" from the Unix special files.- To write zeros to a disk, use
dd if=/dev/zero of=/dev/sda bs=16M. - To write random data to a disk, use
dd if=/dev/urandom of=/dev/sda bs=16M.
The option makes dd read and write 16 mebibytes at a time. For modern systems, an even greater block size may be faster. Note that filling the drive with random data may take longer than zeroing the drive, because the random data must be created by the CPU, while creating zeroes is very fast. On modern hard-disk drives, zeroing the drive will render most data it contains permanently irrecoverable. However, with other kinds of drives such as flash memories, much data may still be recoverable by data remanence.
Modern hard disk drives contain a Secure Erase command designed to permanently and securely erase every accessible and inaccessible portion of a drive. It may also work for some solid-state drives. As of 2017, it does not work on USB flash drives nor on Secure Digital flash memories. When available, this is both faster than using dd, and more secure. On Linux machines it is accessible via the hdparm command's option.
The shred program offers multiple overwrites, as well as more secure deletion of individual files.
Data recovery
involves reading from a drive with some parts potentially inaccessible. is a good fit with this job with its flexible skipping and other low-level settings. The vanilla, however, is clumsy to use as the user has to read the error messages and manually calculate the regions that can be read. The single block size also limits the granuality of the recovery, as a trade-off has to be made: either use a small one for more data recovered or use a large one for speed.A C program called was written in October 1999. It did away with the conversion functionality of, and supports two block sizes to deal with the dilemma. If a read using a large size fails, it falls back to the smaller size to gather as much as data possible. It can also run backwards. In 2003, a script was written to automate the process of using, keeping track of what areas have been read on its own.
In 2004, GNU wrote a separate utility, unrelated to, called. It has a more sophiscated dynamic block-size algorithm and keeps track of what has been read internally. The authors of both and considers it superior to their implementation. To help distinguish the newer GNU program from the older script, alternate names are sometimes used for GNU's, including , , and .
Another open-source program called uses a sophisticated algorithm, but it also requires the installation of its own programming-language interpreter.
Benchmarking drive performance
To make drive benchmark test and analyze the sequential system read and write performance for 1024-byte blocks:- Write performance:
dd if=/dev/zero bs=1024 count=1000000 of=1GB_file_to_write - Read performance:
dd if=1GB_file_to_read of=/dev/null bs=1024Generating a file with random data
Converting a file to upper case
To convert a file to uppercase:Progress indicator
Being a program mainly designed as a filter, normally does not provide any progress indication. This can be overcome by sending an signal to the running GNU process, resulting in printing the current number of transferred blocks.The following one-liner results in continuous output of progress every 10 seconds until the transfer is finished, when is replaced by the process-id of :
Newer versions of GNU support the option, which enables periodic printing of transfer statistics to stderr.