Storage Sizing Engineering: Decimal SI vs. Binary IEC, File Systems & RAID Parity
An authoritative technical exposition on storage capacity discrepancies, file system formatting tax (NTFS, ext4, ZFS, APFS), and enterprise RAID protection calculations.
1. The SI vs. IEC Dichotomy: Why That 1 TB Drive Reads as 931 GB in Windows
One of the most persistent sources of customer confusion in computing and storage engineering is the discrepancy between the advertised capacity on a hard drive or solid-state drive retail box and the available capacity reported by an operating system. When an engineer connects a brand-new 1 TB NVMe SSD to a Windows workstation, Windows Disk Management immediately displays 931.32 GB. Users frequently suspect a defective drive, hidden bloatware, or partitioned factory restore images. In reality, this gap is the direct mathematical result of two conflicting unit definitions:
-
The International System of Units (SI Base-10 Decimal): Standardized by the International Bureau of Weights and Measures (BIPM), storage hardware manufacturers (Western Digital, Seagate, Samsung, Kioxia, Micron, Solidigm) measure storage using standard metric prefixes:
$$1\text{ kB} = 10^3\text{ Bytes} = 1,000\text{ Bytes}$$ $$1\text{ MB} = 10^6\text{ Bytes} = 1,000,000\text{ Bytes}$$ $$1\text{ GB} = 10^9\text{ Bytes} = 1,000,000,000\text{ Bytes}$$ $$1\text{ TB} = 10^{12}\text{ Bytes} = 1,000,000,000,000\text{ Bytes}$$ -
The International Electrotechnical Commission (IEC 80000-13 Base-2 Binary): Operating system memory architectures, CPU address buses, and low-level kernel paging mechanisms operate on powers of two ($2^n$). In 1998, the IEC codified binary prefixes (kibi-, mebi-, gibi-, tebi-, pebi-) to distinguish base-2 powers from base-10 powers:
$$1\text{ KiB (Kibibyte)} = 2^{10}\text{ Bytes} = 1,024\text{ Bytes}$$ $$1\text{ MiB (Mebibyte)} = 2^{20}\text{ Bytes} = 1,048,576\text{ Bytes}$$ $$1\text{ GiB (Gibibyte)} = 2^{30}\text{ Bytes} = 1,073,741,824\text{ Bytes}$$ $$1\text{ TiB (Tebibyte)} = 2^{40}\text{ Bytes} = 1,099,511,627,776\text{ Bytes}$$
When Windows calculates disk space, it divides the raw byte count ($1,000,000,000,000$) by $2^{30}$ ($1,073,741,824$) to determine gigabytes, yet labels the resulting quotient as “GB” instead of the standards-compliant “GiB”. Consequently, $1,000,000,000,000 / 1,073,741,824 = \mathbf{931.32\text{ GiB}}$. Apple macOS (since OS X 10.6 Snow Leopard) and Ubuntu Linux GUI display drives in true decimal SI units ($1\text{ TB} = 1.00\text{ TB}$), eliminating user confusion, while enterprise CLI tools (lsblk, fdisk, zpool) report IEC units.
Crucially, as drive capacities scale from gigabytes to exabytes, this numerical divergence compounds exponentially:
- Megabytes vs. Mebibytes: $\frac{10^6}{2^{20}} \approx 0.9537$ → $4.86\%$ discrepancy.
- Gigabytes vs. Gibibytes: $\frac{10^9}{2^{30}} \approx 0.9313$ → $7.37\%$ discrepancy.
- Terabytes vs. Tebibytes: $\frac{10^{12}}{2^{40}} \approx 0.9095$ → $9.05\%$ discrepancy.
- Petabytes vs. Pebibytes: $\frac{10^{15}}{2^{50}} \approx 0.8882$ → $11.18\%$ discrepancy.
- Exabytes vs. Exbibytes: $\frac{10^{18}}{2^{60}} \approx 0.8674$ → $13.26\%$ discrepancy.
2. File System Architecture & Inode Allocation Overheads
Beyond the binary-to-decimal unit conversion gap, partitioning and formatting a block device incurs an additional “file system format tax.” A file system must write internal data structures, allocation bitmaps, transaction journals, and inode tables before accepting user files:
-
Windows NTFS (New Technology File System): Formats disks using 4 KB allocation units (clusters). NTFS reserves approximately $1.5\%$ of volume capacity for the Master File Table (MFT) zone, volume boot records, cluster allocation bitmaps (
$Bitmap), and the transactional log ($LogFile). The MFT zone is pre-allocated up to $12.5\%$ to prevent MFT fragmentation, though user files can spill into this space if the partition nears saturation. -
Linux ext4 (Fourth Extended Filesystem): By default,
mkfs.ext4pre-allocates fixed inode tables (typically 1 inode per 16 KB of disk space), consuming $1.5\%\text{ to }2\%$ of the volume. More significantly, ext4 reserves $5\%$ of total blocks strictly for the root superuser (uid 0) to prevent disk fragmentation and ensure system daemons (syslog, sshd) continue running if a disk fills up. On an enterprise 20 TB disk, this $5\%$ default silently locks away 1 TB of usable space unless specifically tuned down viatune2fs -m 0 /dev/sdX. - Linux Enterprise XFS: Employs dynamic inode allocation and high-performance allocation groups (AGs). XFS pre-allocates metadata journals and internal B+ trees, incurring a lightweight formatting penalty of roughly $0.8\%\text{ to }1.2\%$, making it favored for high-throughput big-data targets.
-
OpenZFS (ZFS on Linux): Operates as a pooled, copy-on-write (CoW) transactional file system. ZFS reserves a mandatory slop space of $3.125\%$ ($1/32\text{nd}$ of total pool capacity) through
spa_slop_shiftto prevent out-of-space deadlocks during copy-on-write snapshot deletions, plus 8 KB per vdev label and uberblocks, totaling roughly $4.5\%$ overhead. - Apple APFS (Apple File System): Employs space-sharing across dynamic volumes within a single storage container. Container metadata, object maps, checkpoint areas, and directory catalog B-trees consume approximately $1.0\%\text{ to }1.5\%$ of container capacity.
Legacy hard drives operated with 512-byte physical sectors. Modern drives (Advanced Format) utilize 4,096-byte (4 KB) physical sectors to enhance Error-Correcting Code (ECC) efficiency and areal density.
When partitioning drives, partitions must be aligned to 1 MiB (2,048 sector) boundaries. A misaligned partition forces the drive controller to perform costly Read-Modify-Write (RMW) double-cycles on every write operation, degrading I/O throughput by up to $40\%$ and accelerating flash wear on SSDs.
3. Enterprise Storage: RAID Usable Capacity Calculations
In data center storage arrays, Storage Area Networks (SANs), and Network Attached Storage (NAS), individual physical drives are pooled into redundant arrays to protect against device failure:
- RAID 0 (Block Striping): Provides maximum I/O performance with zero redundancy. Usable capacity is $N \times S_{\text{min}}$. Any single drive failure destroys all stored data across the volume.
- RAID 1 (Mirroring): Duplicates all data across a drive pair. Usable capacity is exactly $1 \times S_{\text{min}}$ ($50\%$ capacity utilization on a 2-drive array).
- RAID 5 (Distributed Single Parity): Uses XOR parity striped across all disks. Survives 1 drive failure. Usable capacity: $$\text{Usable Capacity}_{\text{RAID 5}} = (N - 1) \times S_{\text{min}}$$
- RAID 6 (Distributed Dual Parity): Employs Galois field polynomial dual parity (P + Q). Survives 2 concurrent drive failures, which is essential for high-capacity nearline SAS drives (>10 TB) where multi-day rebuild times create high risk of Unrecoverable Read Errors (UREs). Usable capacity: $$\text{Usable Capacity}_{\text{RAID 6}} = (N - 2) \times S_{\text{min}}$$
- RAID 10 (Stripe of Mirrors — RAID 1+0): Combines striping performance with mirror redundancy. Requires an even number of drives ($N \ge 4$). Usable capacity is strictly $\frac{N}{2} \times S_{\text{min}}$ ($50\%$ storage efficiency).
| Advertised Drive Size | Decimal Raw Bytes | Binary OS Capacity | NTFS Usable (1.5%) | ext4 Default (6.5%) | ext4 Tuned (1.5%) | Primary Enterprise Use Case |
|---|---|---|---|---|---|---|
| 256 GB NVMe SSD | 256.00 × 109 B | 238.42 GiB | 234.84 GiB | 222.92 GiB | 234.84 GiB | OS Boot Drive / Thin Client / Edge Appliance |
| 512 GB NVMe SSD | 512.00 × 109 B | 476.84 GiB | 469.68 GiB | 445.84 GiB | 469.68 GiB | Standard Laptop / Developer Workstation |
| 1 TB Commercial SSD | 1.00 × 1012 B | 931.32 GiB | 917.35 GiB | 870.78 GiB | 917.35 GiB | High-End Client / Small Business Server |
| 2 TB Enterprise NVMe | 2.00 × 1012 B | 1.819 TiB (1862.6 GiB) | 1.792 TiB | 1.701 TiB | 1.792 TiB | Database WAL / Hypervisor Host Virtualization |
| 4 TB Enterprise SSD | 4.00 × 1012 B | 3.638 TiB | 3.583 TiB | 3.401 TiB | 3.583 TiB | SAN Flash Tier / Fast Tier All-Flash Storage |
| 8 TB Nearline SAS | 8.00 × 1012 B | 7.276 TiB | 7.167 TiB | 6.803 TiB | 7.167 TiB | Mid-Range NAS / Video Surveillance CCTV |
| 12 TB Enterprise HDD | 12.00 × 1012 B | 10.914 TiB | 10.750 TiB | 10.205 TiB | 10.750 TiB | Enterprise Backup Target / Video Vault Archive |
| 16 TB Datacenter HDD | 16.00 × 1012 B | 14.552 TiB | 14.334 TiB | 13.606 TiB | 14.334 TiB | Cloud Object Store / Ceph OSD Pool / Gluster |
| 20 TB Datacenter HDD | 20.00 × 1012 B | 18.190 TiB | 17.917 TiB | 17.008 TiB | 17.917 TiB | Hyperscale Cold Storage / CMR Enterprise Grid |
| 24 TB Datacenter HDD | 24.00 × 1012 B | 21.828 TiB | 21.500 TiB | 20.409 TiB | 21.500 TiB | Modern Density Archive (HAMR / Ultra-CMR) |