Idera SQLsafe Followup: Problems with Restoring Databases

It’s now been a month since I initially ran into a few nasty bugs in Idera SQLsafe v4.5, and it’s time to post a followup.

Within a couple days of my original blog entry, “Idera SQLsafe v4.5 Review: Too Many Showstopper Bugs“, I was contacted by staff at Idera. They agreed about the issues, and pledged to correct them in the next build of SQLsafe. They’re great people - I’ve contacted them for support before - and I enjoyed interacting with them.

There was another support issue that I didn’t blog about, because I didn’t want to spread panic (and still don’t.) However, it’s still not fixed, and if I was a DBA shopping for backup products, I’d want to hear about it, so here goes.

I have a two terabyte SAP BW data warehouse on SQL Server that I backed up with SQLsafe v3.1 for a year. I regularly restored the production database over to our QA server several times a year, and the restores always went flawlessly. SQLsafe paid for itself countless times over because it cut my backup size from 2tb under 400gb, giving me a smaller backup window, less tape costs, and the ability to back up straight to a network share. I never had a problem with it.

When we deployed SQLsafe v4.5, I didn’t test the 2tb restores right away, because our developers were working on the QA server and I couldn’t overwrite their work.

On October 1st, I tried a restore, and … it failed. I tried several troubleshooting steps, beating my head against the wall thinking it was a problem on my end (because it usually is).

On October 5th, I gave up and opened a support ticket with Idera. I got one bad apple on their support desk, and his answer was for me to apply a SQL 2000 hotfix. After I pointed out that my support ticket was for a SQL 2005 instance, he asked me to change memory settings on SQL, and ended with this quote (bold emphasis is mine, typos are theirs):

“I would not expect fragmentation to matter too much in 2005, but you van still get a large amount of the memory stuck in allocation. Increasing the memory size should also help resolve this. To do either you will need to restart SQL server I am afraid. I wish I had better news, but this is a SQL server side issue so other then dropping MemToLeave(as you have already done) there is not mush of anything more that can be done on the SQL safe side.”

I was furious: I’d never had this issue before with SQLsafe v3.1, so in my mind, it couldn’t be a SQL problem. The only thing that had changed was Idera SQLsafe.

Tip for IT professionals: if you don’t get the support quality you want, escalate the issue. I knew Idera had solid, professional support, so I pressed and escalated the issue at Idera. Sure enough, I got the answers I needed. Later that day, they admitted that they’d reproduced the issue internally and they were working to fix it. They hoped to get me a repaired build that afternoon, and then that afternoon, they said it would be October 8th before I could get a working build. I said I was willing to take an alpha version to test the restores - even if it crashed my QA server, that would be OK, since this was the only database on the server anyway, and the server was down for the count, waiting on the restore.

On October 8th, they still didn’t have a working version ready.

I was running out of options. I didn’t have 2tb of free drive space on the SAN to do a native SQL backup. I didn’t have any other backup products that could handle it, and I couldn’t just stop the production data warehouse so that I could back up the ldf/mdf/ndf files to tape. The developers were getting frustrated because their server had been down nearly a week, and I didn’t have any good options that didn’t involve spending money. My managers were unhappy (to say the least) at the prospect of buying another backup product when we already owned one, and it was looking like I’d have to downgrade my production servers from v4.5 to v3.1 - live - without a reboot. That made me pretty nervous.

Then, I got lucky: Quest stepped in and offered me free licensing for their Litespeed product. I tried it, and it worked the first time. I dodged a bullet, although my reputation at Southern got wounded. For a while, I was the DBA who couldn’t restore from a backup.

I have to be honest: I wanted (and still want) the Idera product to work. I genuinely like SQLsafe, because the user interface is fantastic. My developers rave about how quick and easy it is to refresh development and QA servers from production. And frankly, we paid for it, and it was installed & working on a lot of servers already - why switch from one product to another if I could avoid it? Therefore, I kept in touch with the Idera folks and said I wanted to test the SQLsafe fixes when they were implemented. As soon as the product could restore my 2tb data warehouse again, I wanted to switch back to it. Nothing against the folks at Quest - Litespeed is great too - but I don’t change horses very often.

On October 17th, Idera let me know that they’d found the bug in a different place than they’d originally expected to find it. I completely and utterly applaud their honesty, and it echoes my usual support experiences with Idera. They’re honest, they’re professional, and they know their stuff. I had the one bad set of interactions on October 5th, but I can forgive that from tech companies because it’s so hard to get great support people. They found a possible workaround (disabling status reporting in SQLsafe) but I didn’t have another 2tb environment that I could use for testing.

So I was running out of time: Idera said, “This will NOT be fixed in our 4.6 release simply because the amount of testing/validation is quite large and we’re still working on the fixes. It will be in 4.7 which will come out as soon as possible.”

Ouch - I couldn’t wait for another regularly scheduled release just to be able to do database restores on my data warehouses. To make matters worse, I had a vacation scheduled at the start of November, and I wanted to have one consistent backup product in place. I had to keep the whole backup/restore thing as easy as possible for the rest of my staff, because I’m the only dedicated production DBA in the shop. While I’m gone, I need the development DBAs and our Windows staff to be able to do restores on demand, and I didn’t want to have to train them on two different products.

So, I switched to Quest Litespeed. I’m not happy about switching only because I’m not happy about any change that I can avoid. I just couldn’t avoid this change. It’s not that I’m disappointed with Litespeed (after all, it saved my bacon and it works great) - I just hate giving up on a product.

On November 6th, when Idera found out that I’d switched, they offered me a private beta version of the 4.6.1 code and promised that it fixed the restore issues. Unfortunately, I was already out on vacation, and it was too late for me to switch back. In their minds, they’ve bent over backwards to accommodate a customer, and they’re right. However, in my mind as the customer, the product didn’t work, and I didn’t get a fixed version until over a month later. I could theoretically put this beta code on my production servers, and it might work fine, but at this point, I can’t afford to gamble any more. When I get back from vacation, I have two days left with an eval LeftHand Networks SAN, and I’ll put those two days into testing the Idera 4.6.1 bugfix. If the restores work, I’ll definitely note it here, because if I was a DBA considering the purchase of SQLsafe, I’d want to know that it works. I hate reading internet posts that leave an open-ended question, leaving me wondering if the product ever worked or not, and I’m not the kind of guy to leave that post hanging without an answer. (Update: we ran into problems with the LeftHand SAN and were not able to test the Idera bugfixes.)

In the coming weeks, I’ll post a series of blog articles comparing the two products since I’ve now got a lot of experience between the two. They’re both great products, they both have their shortcomings, and they’re both worth the money. Shops without a backup compression product don’t know what they’re missing, and it’s not just backup compression; both Idera SQLsafe and Quest Litespeed offer tons of advantages that save me time and money. I’ll demonstrate how DBAs can write an ROI proposal to show how the products really do pay for themselves, all sales BS aside.

Update on February 10, 2007 - I get an email at least once a week whether a new version of SQLsafe fixed the bugs.  We stopped using SQLsafe, so I can’t say, but you can get an evaluation version from Idera to find out by testing in your own environment.