T-SQL Tuesday #44–Second Chances

It’s T-SQL Tuesday time again. This is the monthly blog party, started by Adam Machanic. If you’d like to participate, write a post on the topic and publish it on the second Tuesday of the month. If you’d like to host, contact Adam.

This month the topic is hosted by Bradley Ball and his subject is Second Chances.

Second Chances

We all make mistakes. I ask the question of people in some of my talks, and note that I make mistakes all the time. I do, and while most of them are fairly small and easily recovered from, some aren’t. Some are easy to brush off, and some are very, very embarrassing.

In my career, when I think about the big mistakes I’ve made, I’ve got a few choices. There have been times I’ve deleted all the data in a table.

There have been times I’ve restarted a SQL Server without telling anyone. There have been “fixes” to the system I’ve made without running through the proper channels. However there is one item that stands out for a few reasons.

SQL Slammer

I’d let patching slip. We had a large SQL Server environment (hundreds of Standard/Enterprise versions, thousands of MSDE versions) and while we’d been trying to stay patched, it was a hassle and we let some things slip in the fall of 2002.

I’d gone to the mountains for the weekend to ski with my family. We returned late on a Sunday night to find numerous voice mail messages at home and on my cell phone. I was actually called again while I was checking messages on the way home from a friend at work that said I needed to come in. After dropping off my wife and kids, I headed to the office.

Our analysis of the worm showed that it was wrecking havoc on our network. It constantly bottlenecked the network, and we had shut everything down. For a 5,000 person company, with a central network presence and hundred of software developers, this was not ideal. We were up most of the night, waiting on someone from Microsoft to fly in and help us rebuild the patch that Microsoft had released. Since we’d installed many instances of MSDE in non-standard locations, the patches wouldn’t work.

Our network was down Mon and Tues, but we learned some valuable lessons, like email and file shares weren’t so critical to the business that we couldn’t function for a day. We also learned that allowing developers to ignore patches was a bad idea as the majority of the delays after Mon afternoon were due to MSDE instances.

My lesson? Myself and the other DBA received a stern talking to from our boss about patching. We had delayed some of the patches for a few months, but there wasn’t a good excuse for getting almost six months behind. Microsoft had been sending lots of patches, but we didn’t have a good reason for not getting them installed quarterly.

I still don’t like the every other month patches that MS releases as cumulative updates, and I don’t recommend them, but I have learned the security updates are worth getting installed ASAP.