What I started with isn’t what I ended with
I started off really trying to stay close to budget. When I realized I needed to go over by a bit (storage, mostly), and that not all of my parts would even be here in the calendar year (the M.2), I started to wonder if I got the parts that I should have. This is what I started with.
It was a reasonable build that stayed close to budget, and I was pretty happy with it.
When reality kicked in, this is what I ended up with. That’s all of last year’s budget, and half of this year’s budget. And trust me, I could have figured out other stuff to buy, too.
What changed?
The CPU went from the i7-6800K to the i7-6850K
The motherboard went from the X99-A to the X99-E
Filled up the ICY DOCK with 6 SSDs (but I would have done this anyway)
Got a 1TB M.2
Threw in an 8TB SATA drive for backups
Added a bunch of case fans (seriously, this thing could terraform a small planet)
Downgraded the graphics card (who cares, right?)
Upgraded the power supply from a 550w to an 850w
Most of the changes didn’t add a significant amount to the price. The M.2 was the real killer.
Feeling a little weird
I started thinking about other options. If I knew how badly I was going to go over budget, I probably would have built this from the get-go. Not that I’m at all unhappy with what I ended up with, of course. But I could have made some cool tweaks.
The main difference here is the motherboard. I would have bought the X99 Pro (on purpose, this time), and nabbed a second 1 TB M.2 — though I’m not sure if I would have put together an M.2 RAID, or created TempDB drives on it for my SQL VMs. The world may never know.
Until 2018.
Other differences: the RAM would have to change to be compatible, but I’d keep the same amount. There’s an incompatibility warning on the page for 128 GB of RAM, but Asus documentation says it’s supported. I sent a note into the site owners, but they haven’t updated it yet. This is also the same board/RAM combo that Stack Overflow devs use, and I’m like 50% sure their computers work.
Understimating
What I imagined myself doing is plugging this thing into one of my laptop monitors, and borrowing a keyboard and mouse from another setup to do the initial install and config. After that, I’d just remote in to do my stuff.
That didn’t quite go as planned. For some reason, this thing didn’t work with a laptop monitor. When I plugged it in, the monitor just ignored it, the way I ignore Surgeon General warnings. I ended up having to do the initial install and config on my livingroom TV.
Yes, 55 inches of Windows installer is a bit odd.
I also didn’t take into account little hiccups, like a BIOS update needing some config changes, adding an External switch in Hyper-V breaking networking, etc. There’s a lot of little tinkers you have to do, and moving this back and forth between my office and the livingroom TV is just stupid, especially when my kid is trying to watch cartoons hit each other with hammers.
So yeah, I caved and bought a cheap monitor, and a keyboard/mouse combo. Yes, my desk is going to look ridiculous, but whatever.
So now what?
With everything built, it was time to get things installed. Word of warning, if for some reason you buy the X99-E, you can’t run Windows Server 2016 on it, as of this writing. I mean, you can, but it can’t figure out which driver to use for the Ethernet port, which is a deal breaker. Intel drivers wouldn’t even install; they’d say there were no Intel devices on board to use. Heh.
Using my favorite system info tool, Speccy, here’s the system summary. For some reason it isn’t showing the RAM as being overclocked, but it is. Weird. Whatever.
And of course, CPU-Z shows things spinning at the speed of light.
Next up
Doing some wheel kicking is in order. I need to start setting up my VMs, which aren’t just for SQL. I need a domain controller with AD and all that. It’ll also be time to start parting ways with some of the stuff I work with on my laptop.
Thanks for reading!
Brent says: in the company chat room, there’s a running joke about our desktops being more powerful than a lot of production SQL Servers. Granted, the total cost on this is up over $4k, but with 128GB memory and several terabytes of insanely fast solid state storage, it’s still a screaming deal.
35 Comments. Leave new
Nice setup and post, thanks for sharing. Why so many SSDs?
Did you look at drivers for Windows Server with the 960 Pro?
I’m using Windows Storage Spaces, so the effective size is about 50% of total after redundancy is set up.
Also, because we use the StackOverflow database for many demos, and I want VMs to set up for different HA/DR scenarios, having multiple copies of a 100 GB database fills up drives quickly.
As for the drivers, no. It worked, though. I could install Windows on the M.2, but I couldn’t get the Ethernet to work, that’s why I ended up on Windows 10 instead.
Can i recommend you something?
When you can buy some “upgrades” (read this like “after the GTX 1080 SLI”), buy a RAID Card like the LSI Megaraid or the Areca Cards…
While Storage Spaces works, if you’re using it for Redundancy (i read this like “I’m doing RAID 5-6 on software”), you will find some performance problems while doing some intensive write tasks…
The chipset support for RAID & Storage Spaces, are essentially Software RAID, and have slowdowns while computing the parity for the data, before write it on the storage…
If you go with a specialized card, like the LSI or Areca, they’re more capable of doin this with more speed, because they’ve implemented dedicated processors to do this task…
And i’m talking about something like software raid writes about 70-150 MB/s on the 6 SSD’s, but the specialized card writes the same about 5-6x faster, minimun… if you’re doing something like having up an VM AD Server with 2 cores, and 2 more VM’s with 5 cores, SQL Server & Availability Groups & Sync Replication, your most notorious bottleneck will be the drive speed, because the “parity” calculus is needed before write the data, for both VM’s, the primary doin the job, and the secondary node impacting the data generated in the primary node…
Actually, using two-way mirroring, I’ve found writes to be about what I get on a local SSD, sometimes better.
Writes were significantly worse using parity instead of two-way mirroring — I forget what, but enough to make me break the setup down and start over.
New build:
Laptop (exact same SSD alone):
I see now your storage is not the reflected in Speecy (apparently)… that explains all xD
But yeah.. the reason why your write speeds were soo bad was the effect of the software RAID doing the parity operations…
From what i’m reading, Two Way Mirroring is something like RAID 0+1 (2 groups of 3 drives, mirroring each other)
The write speeds are a little conservative i suppose… i was expecting something near the Gb/s, like the read speed (PS: i have the same ssd in my computer)
It’s in Speccy; it’s the last line (Storage Space Device).
Yeah, but not the same capacity D:
If Two Way is like RAID 0+1, 3 x 894 = 2682 Gb aprox… in line with your CDM Screen (F drive, 2673 Gb size)
Speecy show the size like it was RAID 6 (4 disc data, 2 parity)… 4 x 894 = 3576 Gb… in line with Speecy screen (3564 Gb)
Oh, yeah, I probably took the cap the first time around when I tried parity. I went with it because it got me a little more space.
Well, now, ending the “recomendations”, it’s really a great build Eric… it’s good to see someone is working on that kind of beast system…
In contrast, i’m currently writing my code on a Core i5 2500 with 8 Gb of RAM at work, who screams “KILL MEEEEEEE” everytime i’m doing some intensive tasks, courtesy of his 5400 rpm 500 gb hard drive… lol… xD (sorry if my english sucks… this response wasn’t sponsored by Google Translate (like the previous posts >.< )
It’s Eri”K”, sorry for misspell your name… >.<
At least you didn’t call me Brent 😉
Nice post Tara.
Why did you go with a 6850k instead of the 6800k? Seems like an extra $150 for almost no gain.
If you check the comments of my other posts on this, there’s a discussion about it.
If i’m not wrong, the only difference between the 6800 & 6850 are the extra PCI Express lanes enabled with the 6850… if you want to do some SLI or Crossfire in the (near) future, it’s goin to help (cof cof Brent you don’t read this cof cof)…
I readit someone said something about “full storage performance”, but in reality, it’s on the PCI Express slots where the extra lanes are going, or in extra NVMe ports on the motherboard
If, in the future, you exchange your mobo for other with support for 2-3 NVMe drives, the extra lanes are going to be shared / substracted from the others PCI Express slots on your mobo (example, your processor have 40 lanes, your mobo have 3 PCI Express slots, 16x 8x 8x… that’s 32 lanes.. 4 lanes are going to the chipset to be shared between the gbit lan, sata ports, and usb 2-3-3.1 ports, and finally, 4 more are exclusive for the NVMe slot 1… if you put 1 or 2 drives more, you would probably ended up with a config of 8x 8x 8x on the PCI Exp slots, because you can’t substract lanes from the chipset)
I’m recently returning again to the “PC Master Race” (about 2-3 years using gaming laptops), so, i’m a little off of touch with the latest details of the HEDT segment from Intel…
Ah, you took my advice and went with a lower-end fanless graphics card… if I’d known you were going to take that route I would have sent you a barely used Radeon 5450 I’ve got laying around.
The unrecognized on-board ethernet issue with server operating systems is a pain, that’s the reason I usually get an Intel PCI NIC for around $30 and just use that, eg. https://www.amazon.com/gp/product/B001CY0P7G/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1
Oh, sure, the Radeon 😉
You know, I was just plan beleaguered by the time I ran into the Ethernet driver issue. If I were a little less impatient, I probably would have done the same thing.
Hello Erik,
I have built a similar system: http://www.sqlpassion.at/archive/2017/01/13/how-i-designed-my-mobile-data-center/
The main difference is that I can take it with me – and YES, the Samsung 960 PRO M.2 disk is *awesome*!
Thanks,
-Klaus
I saw that! That’s really cool. If I got out of the house more, I probably would have wanted something portable, too 🙂
M.2 is super fast but the problem is motherboards have so few slots.
Check out this board for holding 4x M.2 devices.
http://amfeltec.com/products/pci-express-carrier-board-for-m-2-ssd-modules/
I get 6000 MB/s seq read using 4 of these
http://www.microcenter.com/product/465274/RD400A_256GB_PCIe_NVMe_M2_Solid_State_Drive
Be careful your 16 wide slot is really getting 16 lanes.
Interesting. Mind if I ask how much yours cost, since there’s no price listed?
About $370 USD.
The link goes to an x8 PCIe but they have an x16 that you want instead.
Each M.2 wants 4 lanes.
My motherboard has four x16 PCIe slots, but only one actually gets 16 lanes.
Yeah, that’s a cool idea, but I’ll probably wait a bit before making any changes or upgrades to this.
Hopefully in the next year or two, motherboards with more M.2 slots will be available, and the price of larger chips will come down.
Why doesn’t anyone consider AMD Threadripper or Ryzen 2 CPUs for these servers? The price is almost half, the number of cores double, you can install more RAM… so why?
AMD still has some trust to earn back: https://www.tomshardware.com/reviews/amd-ryzen-3000-turbo-boost-frequency-analysis,6253-4.html
There’s a lot of bad journalism out there right now about the AMD products. The technology and node is quite new. There’s all sort of teething issues but nothing too bad. For example, there’s Turbo Boost 2.0 and 3.0 with Intel while with AMD there’s PB, PB 2, XFR, PBO. AMD is closer to the new era of less user-overclockable CPUs, everything is automatic now. With PBO and XFR, if the CPU can stay below it’s max temperature and power draw, it can automatically overclock well above it’s rated TDP. But that assumes you have a very beefy cooler and good VRM. And then there’s the issue with the chiplet design which forces the issue with cooler mounting pressure and bad coolers, because now the chiplets are more spread out towards the edge of the CPU, as opposed to previous monolithic designs where the heat originates from the center of the CPU which meant that a little convex CPU heatspreader would mean better contact with the heatsink, which is actually how most Intel CPU heatspreaders are. https://www.gamersnexus.net/guides/3491-explaining-precision-boost-overdrive-benchmarks-auto-oc
Tom’s Hardware is bad journalism? Alrighty then, we’ll agree to disagree. Thanks for stopping by though!
I’m grateful you could have me! But yes, unfortunately TH isn’t exempt from some pieces of bad (objective, 100% unbiased, correct) journalism, albeit by far it’s not the worst either. I don’t 100% believe any piece of journalism I read. I generally average over multiple sources. Big and small. But TH has had some ups and downs (https://www.tweaktown.com/news/62951/internal-struggles-toms-hardware-go-public/index.html and https://www.tweaktown.com/news/62960/toms-hardware-founder-comments-rtx-just-buy-it-scandal/index.html) and lots of people pointing out their bias towards Intel and NVIDIA, their posting of some unchecked facts, their apparent higher interest towards making more money rather than interest towards readers and technology users in general. Like I said, I average… I take the median. When I said there’s a lot of bad (unchecked, early, hastied) journalism out there, I didn’t specifically target TH though. I was referring to the early stuff that is available right now. My (median) take is that AMD has a better product than Intel at the moment and even pros like you should consider it. I realise you don’t have all the money in the world but maybe you can hopefully find someone to lend you some Threadrippers. 🙂 – I would look forward to reading what you would have to say. There’s very little EPYC/Threadripper vs Intel for SQL Server information out there right now.
SQL Server’s license prohibits you from publishing benchmarks without Microsoft’s approval, so I wouldn’t wait around for bloggers to work on that.
On the trust or not on Tom’s
Actually, Tom’s has been having a really good amount of “perlitas” lately (like the “just buy it” article Paul linked)… isn’t bad journalism per se, but has been cuestionable lately on some fronts, with some conflict of interest for very much people…
I tend to read/see youtube channels like Anandtech, Gamers Nexus, Hardware Unboxed, Level1Tech, Linus Tech Tips (for the non tech savyy people, its one of the best sources of info)
About the benchmarks
I think the license focus is more about comparisons between SQL Server and other RDBMS… but using the same version of SQL Server in an Intel vs AMD CPU test to show the CPU performance (like using the 300+ GB Stack Overflow DB, and then run some heavy querys on cold/warm conditions, reading the time statistics), could be done with their approval (you have more years of experience, so, probably something about the benchmarks passed over my head, but that’s the way I read it)
Right, *with their approval* – that’s the hard part. They want to see in detail the testing methodology that you used, and have their input in changing the tests where necessary. I simply don’t have time to do that level of testing for free. Maybe you do – by all means, go for it! I’d love to read it too.
Yes Luis, that’s the same feeling I’ve been getting lately about Tom’s from some of the very same sources you mentioned and some occasional browsing of their articles.
And yes, I would love to see CPU benchmarks, AMD vs. Intel / Intel vs. AMD, specifically on SQL Server workloads, if licensing permits (we’re not talking about benchmarking SQL Server here, but the CPUs). I’d be interesting in building my next servers with AMD CPUs if they offer better performance for the buck.
In other news, AMD just came out with the Zen 2 cores which are on par with Intel on instructions per clock or something like 1-2% behind. Based on Zen 2, there’s consumer chips called Ryzen 3000 series (Ryzen 2) of which the Ryzen 9 3900X offers 12/24 Cores/Threads for $500 that clock just like the $500 8/16 C/T Intel 9900K. There’s also a $700-750 Ryzen 9 3950X with 16/32 C/T. Epyc Rome would be the chips that you would have to compare to the Xeons but those are not here yet, but will be, this year, I think August. Those would go up to 64 cores. 128 Threads! :O
Ironically, here’s an early article from Tom’s with a table with Zen 2 AMD processors and presumed pricing: https://www.anandtech.com/show/14525/amd-zen-2-microarchitecture-analysis-ryzen-3000-and-epyc-rome
Plus with Zen 2 you get doubly-faster IO with PCI-E 4.0 and more lanes (for which SSDs are just starting to appear and they – are – blazingly – fast! :D). Or you could get the 2nd gen Zen 1 core based Ryzen Threadripper 2950X with 16/32 Cores/Threads but those are like 25% slower than Zen 2.
Here’s a more recent leak on the upcoming Epyc line-up https://www.extremetech.com/computing/293544-amds-entire-epyc-rome-product-stack-just-leaked
To see why I’m so excited about AMD, check out the leaked benchmarks of the ~$750 Ryzen 9 3950X 16C/32T CPU beating Intel’s $2000 flagship Core i9-9980XE 18C/36T CPU in both single-threaded and multi-threaded workloads: here’s Tom’s again https://www.tomshardware.com/news/amd-ryzen-3950x-vs-intel-i9-9980xe-geekbench,39640.html and here’s UFD Tech’ cover https://www.youtube.com/watch?v=CtnVivgxJqI
And this the last I’ll write about this here. I’ve spammed this comment section enough.
Paul – thanks for hitting the brakes. For further discussions about CPU, you’re best off discussing at a PC enthusiast site.
I realized I kept refering to Threadripper but that would be the HEDT platform. Perhaps I should have said Epyc. But I doubt you’d ever have that kind of CPU in your development box. It’s way too expensive.