The 10X Programmer

Is there such a thing as a 10x programmer? It’s an interesting idea, and one that is discussed from a few perspectives in this Q&A. There’s some talk about how to define “x” as well as the idea that the environment and project might make a big difference in productivity. Certainly the baseline of the staff measured makes a difference. If most of your programmers aren’t very productive, then a more talented developer might easily outperform the average by a factor of 10. Depending on how you measure productivity.

I wonder, though, is there a 10x DBA or database developer? What would that look like? If we examined the effectiveness of a DBA managing systems, I have seen DBAs that managed 10x the absolute number of databases and instances. Working inefficiently, connecting the individual systems to apply updates or check statuses could easily make you ten times less efficient then the DBA making use of scripting and the ability to apply changes across many systems, not to mention managing them for issues.

In terms of database development, if we have some programmers that can produce code using 10x less resources than others, I’d consider that a 10x programmer. Given the state of questions I see asked on SQLServerCentral and other sites, it definitely seems that there are plenty of developers being paid to produce poorly written T-SQL code. We don’t have great definitions of how we can measure productivity, but there is plenty of code posted that can be dramatically optimized. It’s the rare week that I don’t see someone noting they’ve reduced the runtime of some code from hours to minutes. Isn’t that 10x more productive?

Ultimately one of the amazing things about software is that we aren’t bound to a particular structure once it’s created and used. We can quickly, and easily, change the way the code is built so that it works differently, does more, runs efficiently, or really anything we want. Of course, testing that the code produces the desired effects and deploying that to different environments in a way that minimally impacts clients is a challenge, but a challenge many in our industry work on constantly, seeking improvement.

I think there are definitely 10x programmers, but that really depends on your situation and how you utilize the people working for you.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 4.2MB) podcast or subscribe to the feed at iTunes and Mevio.

Posted in Editorial | Tagged | 2 Comments

Why We Need Backups First

One of the things I show when I talk about TDE is that the backup of the certificate is very important. When you create a certificate, SQL Server warns you right away that a backup of the certificate is needed. Following Murphy’s Law, if you continue on with enabling TDE, right after the encryption is completed, your master database will become corrupt and you won’t be able to get to your data. It just takes a few seconds, so back up your certificates.
I was reminded of this while reading a short piece on some police body camera footage being lost. I don’t want to focus on the police actions or cameras or anything but this statement: ” A backup system had been purchased but hadn’t been set up. “
That’s a fairly egregious breach of duty for some IT personnel in this case. The system was installed in 2009, and this event apparently occurred during a software upgrade in 2012. While I understand a backup system might not get configured immediately, I think 3 years stretches the limit of an acceptable delay. However, even if the backup system were not in place at the time of the software upgrade, wouldn’t a backup before upgrade be required? Isn’t that a standard practice?
There have been other reports of similar issues, which is disconcerting. It appears the use of technology for archival of data isn’t well developed, or properly understood by the IT staff in charge of the process. Of course, this happens in plenty of enterprises and other organizations where IT staff seem to think that backups are important in and of themselves. In case you’re confused, they are not. Restores are the important things to have, which means you must verify your backup process allows the data to be retrieved.
Many of us have scripted methods that practice this regularly, refreshing other environments, restoring databases for DBCC, or some other task. Few of us actually practice this ourselves across all databases on a regular basis, ensuring our skills for tail log backups, NORECOVERY and STANDBY restores are fresh and ready for emergencies. However, we should not take this task lightly. It is important that we are able to recover data in almost all situations.
If we can’t do that, then we are truly disregarding our duty as the custodians of data.
Steve Jones
Posted in Editorial | Tagged , | Leave a comment

The Danger of xp_cmdshell

Securing a computer is a challenge. There are all sorts of potential issues in every platform, and ensuring safety for your data can be less a reflection of your ability and more the good fortune there isn’t a focused effort to attack your systems. However, we certainly also face issues with inside users, many of which may make mistakes that are accidental more than malicious. It’s for these reasons that we look for secure by default applications and a reduced surface area for any system.

Many people refuse to turn on xp_cmdshell as an option for scripting in SQL Server. This is disabled by default, and quite a few DBAs are glad of this setting. However, there are plenty of people that think xp_cmdshell isn’t a big security risk. There are certainly ways to mitigate the usage by non-privileged users, and this can be a tool that is very handy for accomplishing work without a lot of development time.

This week, as security issues become more important to us all, I’m curious how you feel.

Do you think xp_cmdshell is dangerous?

I have to admit that I’m torn. I don’t think this inherently dangerous. It does open up some attack vectors, but the last few versions of SQL Server have allowed some limitations, so I would enable this if needed to solve some issues without too many concerns. However, I wonder if many of you feel the same way.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.2MB) podcast or subscribe to the feed at iTunes and Mevio .

Posted in Editorial | Tagged , | 4 Comments

Data Science Education

One of the good skills to have for a data professional is how to analyze data. Most of us could learn more about data science and data analysis for some aspect of our jobs. We are data professionals, so we should understand how to analyze data. I’d expect that a competent data professional would be able to put together a report on some set of data that means something to an end user. If we administer systems, then analyzing usage (index, space, etc) is a skill we need. If we write code, sooner or later we’re going to write some report for a client. Either way, we need to perform some sort of analysis.

How can you learn more? There are lots of resources available. I thought it was interesting that Microsoft has teamed up with edX for their own data science degree. While there are mixed feelings on this, I think edX has a good platform and strong partnerships for teaching. There are other places, such as Coursera, that are doing the same thing, offering a variety of courses online.

In fact, it appears that quite a few educational institutions and businesses are starting to increase their data science related offerings. There’s a good summary of some of the options in a piece from Dataversity. In response to all the demand, or at least perceived demand, there is everything from a boot camp getting you up to speed on some quick analytic techniques and tools to full degree programs. Some programs have different levels, depending on the amount of statistical knowledge you have.

Learning something about statistics is probably the best way for most people that would like to get started. I’ve seen quite a few people start learning by reviewing some statistics techniques. From Buck to Mala, putting your own learning down in a blog will force you to ensure that you actually understand the principles. As for getting started, there are lots of great books you can buy, or read information online. One note, be careful and double check the information written on random blogs. Or check out places like the Khan Academy.

In some sense this reminds me to the rush to get a CNE in the early 90s, or the many boot camps that appeared to help people a MCSE certification in the mid to late 90s. Lots of people passed the tests, but weren’t very qualified, which has contributed to the general distrust of certifications today. I hope that doesn’t happen too much today, though I’m sure it will to some extent. The chance to make more money, or just find a job, will drive lots of people to look for quick wins.

While I’m sure some of you will get better jobs, perhaps making a lot of money with minimal data science experience, I hope many of you continue to learn and improve your skills in the data analysis area over time, whether you are paid to be a data scientist or not.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.5MB) podcast or subscribe to the feed at iTunes and Mevio .

Posted in Editorial | Tagged , , | Leave a comment

Quick #SQLPrompt Tips – Expanding Wildcards

I tend to try and get code on the screen quickly and then start to remove things. I’m a visual person, and it’s helpful for me to see some tables, joins, filters, and columns as I’m structuring a query.

One of the ways I work quickly is with SQL Prompt is that I’ll write a query, using the SELECT * to hold the place where columns will appear. Since I’m not always sure what columns exist in a table, using the asterisk allows me to complete a valid query.

2016-08-26 08_46_03-30113.sql - (local)_SQL2014.AdventureWorks2008 (PLATO_Steve (73))_ - Microsoft S

However, I don’t want to leave the asterisk there. Let’s put the cursor behind it. As you can see here, a tip pops up.

2016-08-26 08_53_16-30113.sql - (local)_SQL2014.AdventureWorks2008 (PLATO_Steve (73))_ - Microsoft S

When we hit Tab (or your completion hotkey), the entire column list expands. All columns, from all tables, qualified if necessary, according to my SQL Prompt settings.

2016-08-26 08_53_25-30113.sql - (local)_SQL2014.AdventureWorks2008 (PLATO_Steve (73))_ - Microsoft S

Now I have a well written query, or if I don’t need all columns, I can easily remove those that I no longer want to retrieve.

This is a quick tip, one that doesn’t do a lot, but has the potential to make developers really think about all the data being returned in large queries with a SELECT *.

Give this a try the next time you find yourself writing a SELECT * query and then remove the columns that you really don’t need. You might also check out a similar piece I wrote for the Redgate blog.

If you aren’t a SQL Prompt user, then think about downloading an evaluation and becoming a more efficient T-SQL developer.

You can see a complete list of SQL Prompt tips at Redgate.

Posted in Blog | Tagged , , | Leave a comment

Down Tools Week 2016

Last week I went to the Redgate Software office in Cambridge, UK. I travel there a few times a year to meet with product groups and touch base with the other people in marketing. However, this trip was planned around Down Tools week, which is an event that Redgate has once or twice a year. This is similar to what other companies have done, like Atlassian ShipIt day, and I had the chance to participate a bit in one of the projects. It was quite fun, and a memorable experience.

The idea is that a project is pitched as an idea for a single week. These are usually ideas that aren’t worth funding as a large project, or would help the world somehow. A team comes together for a long week and has to showcase their work by Friday afternoon. There have been projects just to try something fun at Redgate and investigate something. We had a number of projects, including a charitable image recognition project for Waterscope. That one was really interesting, as some of the software and documentation improvements that were made will be pitched to their investors and taken our for field trials.

I got involved with the Rescue DLM Dashboard project. I like DLM Dashboard as a tool, but it needs some work and should provide more value. A team got together with the idea of seeing where we could add more value and make this a commercially viable product. We also tried to fix a few bugs and get some UX love for the tool. By the end of the week, we had integrated DLM Dashboard with a couple other projects, and had other items to work on. We did win a couple of the contests (best t-shirt, best presentation), but we still have a commercial brief to write and get approved before any more work will be done.

The project structure itself was interesting, with a daily standup at 11am, and teams of programmers working in pairs to add features. I didn’t do any coding, mostly because my C# skills are far below others, and I had other commitments during the week. I was in and out of the dedicated conference room, talking with the project managers and watching developers work through the coding. We had a few interns that worked with experienced developers, and it seemed that people worked well together, sharing ideas and solutions for issues.

I was impressed that the setup of everyone’s workstations, all moved to a conference room, connected, and with cloned git repos was done fairly quickly, with working builds for most people by Monday at lunch. It’s not as simple as one might expect to grab a new project and get a working build, especially on a complex piece of software, and it was fascinating to watch people debugging issues across web pages and local services. I was also pleased to see how open other teams were to lending us a person for a day or two in order to facilitate integrations or extend APIs.

Down Tools week is expensive, but it certainly could be done in different ways. An organization wouldn’t need to cater food every night. Pizza or other alternatives might be fun for some groups. However, I think this can be a great way to create some excitement for your developers, as well as investigate some research that might not otherwise be feasible to undertake. I don’t know that you need to make as big a production as Redgate does, but I’d encourage you to think about taking a week off from normal projects once a year and letting developers work on things that might excite them at your organization.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 5.7MB) podcast or subscribe to the feed at iTunes and Mevio .

Posted in Editorial | Tagged | Leave a comment

T-SQL Tuesday #82–Azure SQL Database

tsqltuesdayIt’s T-SQL Tuesday time and this month we have Jeffrey Verheul giving us the topic of the cloud, or Azure SQL Database. It’s a good topic, given how many people are talking about working in the cloud. There are also the tremendous resources that Microsoft is investing in Azure, so cloud database posts are a good thought for this month.

Whether you are writing today (Sept 13) or at some point in the future, it’s fun and informative to participate in the T-SQL Tuesday blog party. Learn something and write about it.

Azure SQL Database

I have done a little work with Azure SQL Database, but not a log. I have a few ideas, but finding time and focusing has been a challenge. When I saw this topic I had all sorts of ideas, but since I was traveling and busy, I ended up arriving at the deadline without any work done.

However, there was one thing I’ve been wanting to try in Azure, so I decided to make this the post. I wanted to create an Azure SQL Database using Powershell. This is a topic that I think lends itself to better orchestration and development, so here goes.

The first step was a quick Google search, which led me to this post on how to Manage Azure SQL Database with Powershell. That’s what I really wanted to do, so I read a bit. The first step was to download the latest Azure Powershell. I did that and ran an install. This required a reboot, something I rarely like, but I swallowed my annoyance and restarted the desktop.

Once that was done, I ran a couple cmdlets to get connected and check what resources I had. First, I had a login prompt.

2016-09-13 14_10_50-Windows PowerShell credential request

Then I got results. As you can see, I had a few, and the list actually scrolls down quite far.

2016-09-13 14_23_23-powershell

I then tested another cmdlet, Get-AzureRMResourceProvider, to get a list of regions.

2016-09-13 14_25_22-powershell

That worked, so let’s create a database. I used the New-AzureRMSqlDatabase cmdlet, which worked well. I provided the resource group name and servername I already had with a new database.

2016-09-13 14_14_44-powershell

That completed, and things looked good. I connected to the Portal, and sure enough, I had a new database.

2016-09-13 14_15_23-dkranchapps - Microsoft Azure

Of course, an S1 will eat up all my credits, so I changed that right away.

2016-09-13 14_15_57-TSQLTuesday - Microsoft Azure

That was pretty easy. I can see how I might now string together the creation of a database with some other work, perhaps adding data, certainly using the Redgate DLM Automation or ReadyRoll scripts to create objects, and then I can run things and tear them down.

Speaking of which.

2016-09-13 14_32_24-powershell

Posted in Blog | Tagged , , | 1 Comment

Bringing Database Devops to DevConnections

Another alliteration post. Maybe I’ll make all my titles tongue twisters to test your talents?

This is my new favorite session. I’m slated to deliver this at a few different conferences this fall, and hopefully you’ll come see it in Las Vegas at DevConnections on Oct 10-13.

The talk is called Bringing DevOps to the Database, and this presentation is designed to show how many of the software development processes that we take for granted with .NET, Java, and other front end applications can be applied to the database. I’ll cover how version control, Continuous Integration (CI), and Release Management (RM) can be applied to database changes.

This has been a focus of mine as an evangelist for Redgate Software, and I’ve found a number of ways that you can adopt common, best practice software engineering in the database development process. I’ll show how Redgate makes this easy, but I cover the process in a tool independent way. You’ll learn what needs to happen. How you get that working is up to you, but if help is needed, Redgate can cover that part.

If you’re looking for some training, some inspiration, and are a full-stack developer, or multi-technology IT worker, think about coming out to DevConnections. There are a slew of technologies being covered (VS, ASP.NET, C#7, Azure, PowerShell, AWS, SQL Server, Sharepoint, Exchange, Hyper-V and more).

Register today, use 500SPKR to save $500, and I’ll see you in Las Vegas this October.

Posted in Blog | Tagged , , , | 2 Comments

Encryption Keys Matter

Perhaps the importance of protecting encryption keys is even greater than we realize. It appears that the NSA and the US Government have been able to read encrypted traffic for some time on the Internet, perhaps for most of the last decade. There was an exploit in Cisco PIX routers, which I’ve used in many companies, that allowed the NSA to gain the encryption keys used by VPN traffic. In fact, it is likely the NSA could actually penetrate the networks on either end of the VPN with those keys.

Some of you may be against back doors for governments in encryption products, and some of you may be for allowing governments access with legal protections. However, the most disturbing part of this for me is that no organization knew their communications, supposedly secure, were being intercepted and read. The nature of the digital world is that exploits can copy information without the holders of that information being aware of the effort. This is much harder with physical items, where the movement of an object from one person to the next is easily noticed.

One of the very difficult things with protecting the data in our databases is trying to understand when someone has actually retreived information they shouldn’t. This is much more difficult than just tracking changes to data, which is more straightforward. We have auditing mechanisms that easily track changes to data, though most of us don’t have this set up or configured to catch all changes. In practice, that might be good enough to prevent data quality issues, but it doesn’t necessarily protect data from read disclosure.

Monitoring what information is accessed is far harder than tracking changes. Do you know if someone in Sales is accessing a single row with customer details or the sensitive information for all customers? Can you tell when a request is legitimate for an application or if there’s an unusual query that might be seeking massive amount of data for download? Those are hard questions, and ones that I think can only be handled by a large amount of activity monitoring along with machine learning assistance to look for patterns in user activity. Other features like Row Level Security can help limit the inadvertent mistakes made by developers or users, but not necessarily prevent a single user, especially a malicious user, from querying information.

Apart from the activity issues, we should ensure that where it is possible, we should be using encryption to prevent accidental disclosures outside of our applications. I think Always Encrypted has possibilities for the database, but the key management for this, as well as key management for VPNs, disk encryption, and other protection mechanisms, needs both more maturity and the open disclosure to prevent back doors from being included in products. We also need more maturity in our software development that takes the implementation and protection of encryption mechanisms seriously.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.7MB) podcast or subscribe to the feed at iTunes and Mevio .

Posted in Editorial | Tagged , | 4 Comments

Bringing DevOps to the Database

Abstract:

DevOps is a hot topic in today’s software development world. However most of the knowledge and experience with DevOps is based around application software and ignores the database. We will examine how the concepts and principles of DevOps can be applied to database development by looking at both automated comparison analysis as well as migration script management. Automated building, testing, and deployment of database changes will be shown.

Level: 200 – Not to technical, but you should have some idea of how software development works to get something out of this talk.

Goals:

  1. Conceptually understand how DevOps principles can easily be applied to the database in either state based or migrations based changes.
  2. Learn how databases can be automatically built and tested in a CI/CD environment
  3. Understand how database releases can be automated, but are different than those deployment/releases made for other software.

Demos:

This presentation has a few demos where I show how I stored database code in a VCS, perform an automated build, and use a scripted release process to move code from one database to the next. The demos are pre-built, so I just show how they work. I can do this with:

  • git, VSTS Build and VSTS Release
  • git or SVN, Team City, and Octopus Deploy

Downloads:

Bringing DevOps to the Database.pptx

Posted in Presentations | Tagged , , , , , | Leave a comment