Enter SANdman
When people buy SANs, it’s often quite a large investment. Whether it’s all SSD, all flash, or there are tiers of storage that different types of data live on, those disks aren’t cheap. When people visualize their SAN, it’s usually just the server and the pool of drives.
But there’s some important stuff in between the Server and the SAN — SAN doesn’t stand for Storage Abstraction Nerglefwomp.
It’s Storage Area Network, and the “Network” part is what causes a lot of problems.
Those wires, especially when they’re attached to a VM Host, have to manage traffic for a lot of different things at once.
It’s really easy to overwhelm even Fibre Channel wiring, when it’s either single-pathed, or when enough concurrent activity hits a multi-pathed setup.
Round The Way SAN
Talk to most people setting up a new SAN, and they’re (hopefully) going to be using 8-10GFC (or Ethernet).
But if you’re moving hundreds of gigs, or terabytes across those wires, you better be darn sure you’ve got plenty of bandwidth. Let’s take a best case scenario, where you’re moving 1 GB across 10 GFC unhindered. It’ll take 800 milliseconds.
But there are 1000 GB in 1 TB, which’ll take around 13 MINUTES.
Let’s use my new hard drive as an example. I’ve got a VM with SQL Server running on there.
When I use Crystal Disk Mark to benchmark my storage, the results are pretty okay for an external SSD.
I’m doing around the 4GFC mark for sequential reads and writes.
But if I run a workload that generates a lot of I/O for various reasons, and re-run the benchmark, things tank significantly.
Sharing Ain’t Caring
When multiple requests all had to go across my USB 3 connection, my speeds got cut in half, or much worse.
Now take some time to think about how many things you’re asking your storage networking to handle concurrently.
- Think about when you have stuff like backups, checkdb, and index maintenance scheduled.
- Think about when you’re doing ETL, or any bulk data activity.
- Think about if you’ve got AGs or Mirroring set up.
When you ask the SAN admin to take a look at the storage dashboard to see why things are slow, they’re not gonna see a blip. Data simply isn’t arriving at the disks fast enough for them to be burdened.
If you had a bar with 100 people in it, 1 waiter, and 10 bartenders, that one waiter wouldn’t be able to take orders to the bartenders fast enough to keep them busy. Your bartenders would look bored, but your waiter would be a wreck.
What’s The Name Of His Other Leg?
If you like learning about this kind of stuff, and you’re going to the lovely and talented SQLBits in 2019, come to my precon.
I might even be bar tending.
Thanks for reading!
6 Comments. Leave new
Did I just see Shizuku Edition CrystalDiskMark?
You did indeed, hahaha!
If your SAN admin is decent at all the first place they will look is the fabric. That said FC with block storage is still the undisputed king of data transfer for a multitude of reasons and will be for quite some time. Perhaps you have suffered some poorly designed fabrics for that I am sorry but pretty much any 8gb plus FC network designed to best practices will never end up being the bottleneck. If you have specific configurations would love to talk more
Really? 8Gbps fiber maxes out at 800 MB per second. My laptop’s SSD does 3x that speed, and that’s JUST ONE DRIVE.
Yeah, at my last gig we were able to resolve major issues where Windows and SQL Server beleived they were waiting on disk. The SAN admin had saw no issue on their end. It was only due to a session I’d been to by David Klee that I’d thought to ask about the cabeling. Turned out only 1 of 4 of the fiber channels were connected.
Interesting – I’d put dual 40Gbps Ethernet NFS over dual 32Gbps FC block for VM SQL Server any day of the week for both performance, flexibility, and cost.
That said, the pipe can *always* be the limiter, especially with modern storage arrays. Dual 8 FC’s to storage wouldn’t even begin to cover my daily workloads, though it might for individual physical SQL servers. Virtualized SQL at scale – iffy unless you have a really good workload placement which implies that server builders know current and future performance factors, too.
With any high performance/high bandwidth app, what is most critical is for the all the admins/architects to understand how their work impacts flows up and down the stacks. It’s easy for a storage dude to say no problem until they know who is making the requests, how the request are made, and how they are getting there. Storage guys sit at the bottom so they need to be aware of everything above.
You can never have “too much” bandwidth. As with anything, though, use wisely. It’s easy to waste.