The Remote DBA

I’ll start this week with a question, which I hope some of you answer in the discussion: would you like to, or do you, work at home the majority of the time?

I remember when I worked in a company and needed to leave my desk and walk to a room somewhere to get some work done on a server console. Some of those rooms were cold rooms, which necessitated me keeping a jacket at my desk. I still remember going to a large company that had cables run from the data center to a couple specific workstations near the administrators’ cubes. At one point we installed a remote IP device allowing us to get to the console of any server without having to walk downstairs, or even use RDP, which was just becoming to Windows machines.

That was the end of my visiting servers in person, and since then, the only times I’ve ever really needed to look at a server was when a critical error prevented me from connecting remotely. Even then, at many of the co-location facilities I’ve contracted with, I could call and have an individual go press a power button. These days, with cloud providers and virtual machines, even that is unnecessary.

Those of us that have worked with SQL Server typically understand that we always make a network connection to work with the server. Even when we’re working on the server console, SSMS, SQLCMD, and more all make a “connection” to the database server. Therefore, is it really necessary that we ever work near a particular system?

I’ve been working from home as a telecommuter for about 8 years. My wife worked in technology from home for nearly 20 years. More and more people are doing so, in fact, there was a piece on Fast Company recently that noted more people work from home than ever before. Unfortunately, this study shows that most people end up working more hours, adding a few from home to the 40 or more they spend at work.

That’s not good, but if we can get some work done at home, why not more? I know meetings and face to face time matter, especially in some jobs, but more and more I find that lots of people that need time working alone could do a portion, perhaps a significant portion, of their work away from the office.

So this week, would you like to do more work from home (or elsewhere)? Do you want more virtual meetings, more communication over email, Slack, Skype, or some other tool? Let me know.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 4.0MB) podcast or subscribe to the feed at iTunes and Mevio . feed

Triple Check Your Restores

I used to work at a nuclear power plant. Not really in the plant, but as a network administrator in the building next door to the plant. Probably a good thing since I struggled to get to work on time and everyone going into the plant had to go through a metal detector like that at most airports. My tenure might have been shorter if I had been late every day to my desk.

However, there was one thing that got drilled into me by each person I knew who did work closely with the power generation side of the business. Everything was (at least) triple redundant. Not only did they want a backup for a component, or a system, but they wanted a backup for the backup. There was a limit to the paranoia, but in critical places, where radiation was concerned, we had multiple backups. One notable, and impressive, area was the power for the control rooms and pumps. In addition to batteries for short term power loss, there was a large diesel generator for each of our two reactors, plus a third that could take the load if either of the first two failed. Those were impressive engines, each about the size of a very large moving truck and jump started with a few dozen large canisters of compressed air that could spin the crankshaft in a split second.

This week there was a report that the database for the US Air Force Automated Case Tracking System had crashed. Apparently the database became corrupted, which happens. However, the surprising part of this story is that the company managing this system reported they didn’t have backups and had lost some data going back to 2004. They are looking to see if there are copies in other places, which I assume might mean exports, old backups or something else, but the reports make this seem like a completely unacceptable situation. I assume this is an RGE event for a few people, perhaps all of the staff working the system.

I was reminded of my time at the nuclear plant because we had a similar situation. We didn’t lose any data, but we found a backup system hadn’t been working for months. Those days we had a tape drive system that automatically rotated 7 tapes. I think this would last us about 4 or 5 days, so it was a once a week job for one administrator to pull the used tapes and replace them with new ones. We had a system where tapes were used 4 or 5 times before being discarded, and our rotation had a tape being used every 3-4 months. However, the person managing the system rarely restored anything.

One day we decided to run a test. I think this was just my boss giving us some busy work to keep us occupied but in a useful way. When we went to read a tape, it was blank. Assuming this was just a mix-up, we grabbed one of the tapes from the previous day and tried it.

Blank.

At this point, my coworker turned a bit red and started to stress. He was in his 40s, with a family and mortgage. I was in my early 20s and had no responsibility here, but I could appreciate his concern. We frantically loaded tape after tape, even looking at the oldest tapes we’d just received from our off-site provider. None were readable, and most were blank. We nervously reported this to our boss, who had us request a sample of tapes from off-site storage going back over 6 months.

Eventually we realized that we hadn’t had any backups for about 4-5 months. The tape drive had stopped working properly, hadn’t reported errors, but dutifully kept retrieving files and rotating tapes each week, unable to properly write any data. No databases, no email, no system was being backed up.

A rush order to our computer supplier had been placed the first day to get us two working tape drives that we manually loaded tapes in each day, and checked them the next morning. Eventually we replaced the drive in our tape leader and instituted random weekly restores to be sure we had working backups. I’m not sure if the plant manager or upper IT management was ever told, but I’m glad we never had to deal with a hard drive crash during that period.

Backups are something we all need to perform. I note this as the #1 thing a new DBA or sysadmin should perform on systems. However, backups are only good if you can read them and actually restore data. I’ve made it a point to regularly practice restores as a DBA, randomly restoring backups with diffs, logs, or to a point in time. Not only do I test the backup, but I test my skills. I’ve also tried to keep an automated process around that restores all production systems to another server to test both the restore as well as run a DBCC CHECKDB. Corruption can live in databases for a long time. It flows through backups, at least in SQL Server, and this is something to keep in mind.

I’d suggest that you make sure you ensure that your backup plan is actually working by performing a few restores. Build an automated process, but also run some manual restores periodically. You want to be sure that you can really recover data in the event of an emergency.

Steve Jones

The RGE

I first heard this little acronym from Grant Fritchey (b | t). He used it when talking about backups and restores, and I like it. However I realize that I’ve never actually noted what it is, so a short blog to do so today.

An RGE is a Resume Generating Event. This is usually when you make a mistake so egregious that you’ll be packing up your personal effects and exiting the building. If it’s really bad, such as releasing financial or other confidential information, you might be escorted out and someone else packs up your things. I’ve seen it happen, and it will shake you. Don’t do this.

We talk about forgetting about backups, or writing bad code or some important task we often perform as causing an RGE. In my experience, that doesn’t happen too often. Companies usually have a fairly high tolerance for mistakes.

However, that tolerance is usually extended only once. Don’t make the same mistake again. I’d also note that some managers can be very short tempered, and a single, large issue might be an RGE in their eyes.

I don’t usually worry about causing an RGE, but I keep the acronym in mind. Especially when I do something that could affect the core parts of my organization’s business.

Good Stories

The IT horror stories. Many of us have them. The mistakes, the trials, and frustrations, these seem to be the stories we collect. Many of us have a story of deleting data, forgetting a backup, making a security mistake or some other item that perhaps should have terminated our employment. Or maybe it did. In any case, I’m sure most of you have a tale about some mistake.

However, I think we don’t talk enough about the wins we have. The good things we do that actually help people, they appreciate, that get us kudos or even rewards. Maybe there’s something that garnered some unsolicited praise for your efforts. I’m wondering this week, what are your good stories in IT. ?

I hear about good stories all the time from my wife. Her horse training efforts produce immediate effects, some of which are very powerful and moving for her clients. I hear about great things my brother does in medicine. Certainly I also get many very kind words from plenty of you about the help SQLServerCentral provides. But I haven’t had a lot of stories as a developer or DBA. Many of substantial upgrades or improvements I’ve made to applications were tempered with complaints about us “changing something.” At times, it’s felt like a no win situation.

Perhaps my best story is from my time with a large sales organization. We had a demo system that provisioned a set of virtual machines to showcase our products. The complexity came from the fact that each demo environment was 6-8 machines, networked together in a VLAN that allowed them to be identical for each salesperson, yet isolated to prevent network issues. This was in 2001, when tools for managing virtual machines were rudimentary, and things like runbooks didn’t exist.

I helped our administration team go from an Excel spreadsheet, with incredibly complex (and unreliable) macros to an web application. The application allowed them to easily track which environments were provisioned for which salesman, making the entire process much simpler, and more importantly, a multi-user system any administrator could access to provision or teardown demo environments. This wasn’t a large piece of software, and not widely used, but it made work much easier for a few people that were struggling with the constant and ever changing demands of busy salespeople.

What’s your best story? Let us know today.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.2MB) podcast or subscribe to the feed at iTunes and LibSyn.

Great Developers

This editorial was originally publised on March 12, 2012. It is being re-run as Steve is on vacation.

Is a great software developer worth 100 average ones? On one hand I think there are some good arguments that it’s not true. One developer certainly can’t write the amount of code that 100 average ones can. However there’s another way to look at things. A great developer can do things that the 100 will never think of, or never consider. He might not write the code that does as many things as 100 people, but I think a great developer could easily write code that performs a hundred times faster than the code 100 developers write.

That’s why you always have an open position available for a great developer. If one is available, and they rarely are, you hire them if they want to work for you. You can always find things for them to do, and they can make improvements in code that your other 5, 10, or 20 developers will never come up with. I’d make sure they fit in your team and get along with others. You can get less work done if you have someone that is too difficult to deal with or too critical of others. While a great developer can accomplish things that others can’t, or won’t, they can’t do all the work.

Ultimately I think that managing great developers is hard, and they are unlikely to stay with your company for a long period of time. However they are rarely available, and for a few years, they might jump start the evolution of your software, and potentially build something that makes your software great. I’d always have an open spot in my team for a great developer, and hire them as soon as they came available, if I thought they would fit in well with the rest of the team.

Steve Jones

Specialist or Generalist

What would you rather be: a expert specialist in some narrow area of SQL Server or a generalist that’s competent in many areas? The same question could apply to any area of technology, like C# or Windows, or even any subject in which you work. What’s your preference?

Most of us don’t have to make a strict choice. Often in pursuit of deep knowledge in one area, say indexing, the effort will lead to lots of good general knowledge in other parts of the platform. I think it’s important that gain both types of skill, as you work through your career and grow your abilities. I find that there is always more to learn, especially in technology. Even in areas that I think I have deep knowledge about, I continually add to, and refine, my skills.

Brent Ozar had an interesting post on this topic, looking at the maturity of a particular subject area. He noted the more mature some area, the more you might learn towards specialization. I think that’s good advice for your career, though I do think that if you find a deep passion, don’t be afraid to specialize in some area. While you might not get a monetary compensation directly back for the effort put in, I think the reward of learning something you love, and achieving expertise is extremely valuable and important to you as an individual.

As we move through our careers in computing, we find there are always new technologies, techniques, and architectures that we will need to understand. We need to grow our knowledge. In many cases we see older ideas re-packaged and updated for the modern times, but that doesn’t mean skills we can rely on prior knowledge. Instead, you will need to update your skills.

I also think more and more we will be driven to give guidance to others as much as use specific expertise to solve problems. With that in mind, I think whether you choose to be a specialist of generalist, you’ll need to continue to learn more about both implementing technology, as well as communicating your recommendations.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.8MB) podcast or subscribe to the feed at iTunes and LibSyn.

Get Inspired – SQL Inspire 2011 New York

Coming up next week is the SQL Inspire 2011 event in New York City. It’s an event with a number of SQL Server community speakers that are here to inspire you in your career, your life, and with SQL Server. The talks cover a variety of topics, and they look interesting. I am very excited to attend the event, and give my talk as well.

The event is from SQL People, the brainchild of Andy Leonard and Brian Moran, designed to build a stronger community by having us inspire each other. This event is based on the TED events, which are inspirational and informative talks that look to explore new ideas in a variety of topic areas. I have loved watching many of the talks, and while I’m not sure I’m up to the same level of presenting as many of these people, I’m going to try.

Please come if you are in New York on the 12th of November. It will be a lot of fun, and hopefully a very motivational event.

It’s Your Career

It’s your career. It’s something you have to take ownership of and work on. I know that life is busy, and training budgets are tight. That’s one reason we started SQL Saturday; it’s a way to bring a training event and a conference experience to many people.

fun2

I posted this tweet almost a year ago, seeing Brent in a class somewhere, learning and taking notes during some session. It was in humor, but I’m a little serious here. We all have more to learn, and while you don’t need to cram it all in this year, you should be taking advantage of your user group, local events, conferences, classes, even reading something in a newsletter on a regular basis.

Many of us are out here to help. I’ve spoken at 12 events this year, 10 of them free, and will be at another free event this week (SQL in the City – LA). However, you’ve got to make the effort to improve yourself. I , and many others, will try to help you, teach you, but you’ve got to do some work yourself.

Pace yourself, learn at a reasonable rate given the other responsibilities in your life, but don’t ignore this aspect of your career.

PS – If you’re in the LA area, there’s still time to register for SQL in the City and get a free day of training. I’ll also be at DevConnections next week and SQLInspire the week after that.

Build Your Own Event at the PASS Summit

There’s a bunch of stuff scheduled for the PASS Summit next week. You can see some of the events on the After Hours page, but there are a few other events happening during the week as well. Most are private, which is fine for the people that get invited, but what about everyone else?

There are well over 1,000 people coming to the Summit, and the majority of them won’t get invited to private vendor parties, or many other events. However outside of the Gameworks party on Thur night, I have a feeling many of them will end up being “that guy”, spending the evenings alone or even working.

Andy Warren and I have started a few events that are relatively open to anyone. We have our Monday Night Networking event, which is full right because of capacity limits. We also started the SQLServerCentral party, which I run and is open if you have a ticket.

However we think you can start your own event. If you want to go see a band in Seattle, then wouldn’t you want to see if anyone else wants to go with you? Maybe share a cab? Perhaps you’re interested in a workout at a local dojo, or you want to attend the Underground Tour (which is very cool).

If you want to do something, and would like to network, have a little company from people that similar interests, we’d like you to do two things:

  1. Create an event at EventBrite. It’s free if you aren’t charging. Just tell people what you event will cost and let them pay their own way. Put a limit on the number of guests, if it’s appropriate.
  2. Post your event at http://passsummitevents.info/. MVP/MCM Denny Cherry (blog | @mrdenny) has set up the site to track what’s happening at the Summit.

 

That’s it. Post your event and let people sign up.

If you are looking for something to do, check out the page at http://passsummitevents.info/ and see if there’s something you’re interested in. The site will tweet your event, but feel free to publicize it around.

A week at a conference should be a lot of fun, packed with plenty of events for everyone. I hope we can get people to build their own get togethers, for anything from tasting beer at the Tap House to dinner at The Cheesecake Factory to taking a tour of the Museum of Flight.

Whatever interests you, set something up or look for people that have a similar interest at http://passsummitevents.info/.

An Awesome Job

Life is awesome and if you can't see it, read this book.

I recently watched The 3 As of Awesome with my daughter and loved it. It’s a short talk (17 min) from Neil Pasricha, the author of the 1000 awesome things blog. The talk goes through a roller coaster of emotions, at least for me, with a pleasant start, a sad middle, and a positive end. There’s some humor in it, and I highly recommend you watch it.

When we finished, my daughter and I talked about what awesome thing had happened to each of us that day. She said that me giving her 5 extra minutes to play with a friend before we left was great. I told her the crazy, illegal, solo, 5 hit volleyball return I made at her practice that made everyone laugh was the highlight of my day. We challenged the rest of the family to find an awesome thing from that day, and everyone enjoyed it. Since then we’ve made it a point to regular ask what awesome thing happened to each other.

The philosophy espoused in the talk is one I try to follow. I stop and appreciate the cool, little things in my life. A refill of my drink when I’m flying economy class makes me smile. I get a good feeling when I can stop and hold the door for someone behind me. I appreciate the health I have, and the fact I can run regularly. A good cup of coffee, the cat wanting to snuggle up when I’m typing in bed, or the chance to step away from my desk and give my wife a kiss during the day are awesome and amazing things in my life.

Work isn’t always awesome, but I’d like to think that you find little things you can appreciate on a regular basis. Maybe it’s code you write that works. Maybe it’s the knowledge that you saved someone else time because of something you did. Maybe it’s that second monitor that lets the work flow a little smoother. Maybe it’s that burst of bandwidth that made a download smoother. Maybe it’s the comment someone made on SQLServerCentral or on Twitter that grew a smile on your face.

We spend a lot of time at work, and it’s often work. I hope you enjoy your job overall, but even when it’s not the job for you, hopefully you can find and appreciate some little thing that occurs every week that shows the universe likes you.

Steve Jones


The Voice of the DBA Podcasts