VMware, RAID, and Storage Architecture Lessons
Deep lessons from enterprise storage architecture work, including RAID configurations, VMware storage design, and the failures that taught me the most
I have spent a significant part of this year working with enterprise storage, and I want to write down what I have learned. Storage is one of those infrastructure domains that gets overlooked until something goes catastrophically wrong. Network engineers are visible. Server admins get credit when deployments go well. Storage engineers are invisible right up until a disk fails and suddenly everyone wants to know how the RAID array is configured.
This post is a deep dive into RAID, VMware storage considerations, and the architecture lessons I have picked up from building and maintaining storage systems.
RAID: Redundant Array of Independent Disks
RAID is one of those technologies that every infrastructure engineer should understand thoroughly, not just the acronyms but the actual performance characteristics and failure modes. Let me walk through the configurations I work with most.
RAID 0 stripes data across multiple disks with no redundancy. Two disks in RAID 0 give you double the read and write performance, but if either disk fails, you lose everything. I use RAID 0 for exactly one thing: temporary build caches where the data can be regenerated. For anything else, RAID 0 is playing Russian roulette with your data.
RAID 1 mirrors data across two disks. Every write goes to both disks. Read performance doubles because you can read from either disk. Write performance stays roughly the same. You can lose one disk and keep running. The cost is that you lose half your raw capacity to the mirror.
RAID 1 is my default choice for OS volumes. The operating system is critical, the data is relatively small, and the rebuild time when a disk fails is short because there is only one disk to copy.
RAID 5 stripes data across three or more disks with distributed parity. You can lose any single disk and the array continues to operate. The storage efficiency is excellent: with four disks, you get three disks' worth of usable capacity. Read performance is good. Write performance suffers because every write requires a parity calculation and an additional write.
RAID 5 was the default choice for many years, but I have become increasingly cautious about it. The problem is rebuild times. When a disk fails in a RAID 5 array, the array runs in degraded mode while the replacement disk is being rebuilt. During rebuild, every remaining disk is working hard, reading every sector to reconstruct the failed disk's data. On modern large-capacity drives, this rebuild can take hours, and if another disk fails during the rebuild, the array is lost.
This is not a theoretical concern. I have seen it happen. A 2TB disk fails in a RAID 5 array. During the 8-hour rebuild, a second disk that has been running for the same number of hours in the same environmental conditions develops an unrecoverable read error. The array is toast.
RAID 6 addresses this by using dual parity. You can lose any two disks and survive. The write performance penalty is worse than RAID 5 because of the dual parity calculation, but the protection is substantially better. For any RAID array with more than four disks, I now recommend RAID 6 over RAID 5.
RAID 10 combines mirroring and striping. You create mirrors (RAID 1 pairs) and then stripe across the mirrors (RAID 0). The result is excellent read and write performance with the ability to survive at least one disk failure per mirror pair. The cost is 50% of your raw capacity goes to mirroring.
RAID 10 is my choice for database storage. The random read/write performance characteristics match database workloads well, the rebuild time is fast because you are only rebuilding a single mirror, and the failure tolerance is predictable.
VMware and Storage
Running VMware adds another layer of complexity to storage architecture. VMware's VMFS filesystem, the way it handles I/O through the hypervisor, and the various storage connectivity options all affect how you design your storage.
VMFS alignment is one of those things that does not matter until it does. If the VMFS filesystem is not aligned to the underlying storage's stripe boundaries, every I/O operation potentially crosses a stripe boundary, causing two physical I/Os for every logical I/O. The performance impact can be 20-30%. Modern versions of VMware handle this better than older ones, but it is still worth verifying.
Datastore sizing is a design decision that involves trade-offs. Large datastores are simpler to manage but create a larger blast radius when something goes wrong. Small datastores are more granular but create management overhead. I have settled on a middle ground: one datastore per workload class (production databases, production application servers, development, and so on) with sizes that balance manageability against risk.
Storage I/O Control (SIOC) is a VMware feature that helps manage contention when multiple VMs share a datastore. Without SIOC, a VM that generates a burst of I/O can starve other VMs on the same datastore. SIOC monitors latency and throttles I/O from VMs that are causing contention. It is not a substitute for proper capacity planning, but it provides a safety net.
Thin provisioning saves storage space by allocating disk space on demand rather than up front. A VM with a 100GB thin-provisioned disk uses only the space it has actually written to. The risk is overcommitment: if you provision 10 VMs with 100GB thin disks on a 500GB datastore, you are betting that they will not all use their full allocation. Monitor actual usage carefully, and set alarms for when datastore utilization crosses a threshold.
The Storage Architecture Lessons
Working with storage has taught me lessons that apply well beyond storage itself.
Capacity planning is a prediction problem. You are trying to predict future usage based on historical trends and business plans. The tools for this are trending graphs, growth rate calculations, and conversations with application teams about their roadmap. The mistake I see most often is planning for today's capacity rather than next year's. By the time you realize you need more storage, procurement and installation take weeks.
Performance and capacity are different problems. You can have plenty of free space and terrible performance because your I/O pattern does not match your RAID configuration. You can have a fast storage system that is nearly full and performing fine. Understanding which problem you are solving is critical because the solutions are different.
Backups are worthless without tested restores. I mentioned the RAID controller failure earlier in the year that corrupted weeks of backups. Since then, I run monthly restore tests. A random backup from the last month gets restored to a test environment and verified. This is the only way to know that your backups actually work. The number of organizations that run backups religiously but never test restores is alarming.
Storage failures are rarely sudden. Disks do not usually fail without warning. SMART data shows increasing error counts, increasing reallocated sectors, increasing pending sectors. Monitoring SMART attributes and replacing disks proactively, before they fail, dramatically reduces the risk of data loss. Reactive disk replacement (waiting for the failure) is gambling with your data.
The fastest storage is the storage you do not need. Before adding more disks or buying a faster array, ask whether the I/O can be reduced. Caching frequently accessed data in memory, optimizing queries to reduce disk reads, using SSDs for hot data and spinning disks for cold data; these approaches often deliver better performance improvements than upgrading the underlying storage.
Looking Ahead
The storage landscape is changing. SSDs are getting cheaper and more reliable. Storage area networks are becoming more sophisticated. Software-defined storage is an emerging concept where the storage intelligence moves from dedicated hardware controllers to software running on commodity servers.
I have been reading about distributed storage systems like Ceph and GlusterFS that spread data across commodity servers with built-in redundancy, eliminating the need for expensive SAN hardware. These systems are not mature enough for production use in most environments yet, but they represent a compelling direction.
For now, the fundamentals still apply: understand your workload, choose the right RAID level, monitor everything, test your backups, and plan for growth. Storage is not glamorous, but it is the foundation that everything else sits on. Get it wrong and nothing else matters. Get it right and nobody notices.
That is the life of a storage engineer.