Successful Projects

What techniques or ideas allow you to build a successful software project? There’s been no shortage of writing and studying on this topic across the last few decades. So many people expend energy and thought trying to solve the problems of building software at a high level of quality, in a reasonable timeframe, and within some scope that matches the desires of their customers.
I’d like to think most of us succeed, but study after study shows that most parties are disappointed in some way. Developers are under pressure and can’t/don’t built the level of quality they’d like. Management sees software as taking too long and costing too much. End users see too many bugs, though I think that overall quality has risen in the last decade are more proven frameworks are used and developers have gained more experience in newer and better development techniques.
I ran across a post on a test for successful software projects. It’s the Kevin Test for Successful Projects, and it’s based on a combination of the Joel Test and the Rands Test. The idea is not that you implement all of these test items, or do them perfectly, but that you work towards doing more of them than not. There are explanations of what each item in the test means in the various pieces.
I think these are good guidelines for projects, and they’re simple ones. They certainly can apply to database development. However most people don’t really want to spend time on these tasks. However, I think the mundane nature of these items slows you slightly, but forces you to think more about what you’re doing. The gains in quality and velocity, in my mind, come from avoiding rework, avoiding losing work, and avoiding regressions.
Sometimes I think the problems in software development aren’t that we write poor code, but rather that we rush a complex endeavor, being willing to redo work rather than proceeding in a more engineered fashion.
Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.8MB) podcast or subscribe to the feed at iTunes and LibSyn.

Build a POC

I have a few friends that work sales engineers for various companies. They often build a proof of concept (POC) project for various customers or potential customers, which is a great job. They get to showcase how a particular product or technology might work in a situation. We see this all time from lots of companies, including a great POC from BMW shown at the recent //build/ conference.

Sometimes the POC is more mockup than reality, but often this is a way to actually test some software and determine if you should invest more resources in that particular direction. I’ve used this myself, setting up a small project to test items, and I recommend this for others that might want to try something out.

For example, I like the idea of Continuous Integration (CI) for databases, but implementing CI in one of your production pipelines, on a production database is hard. It’s complex, and it can be distracting, not to mention stressful. Who wants to make a deployment mistake in production?

What I recommend instead is that you take a small database and practice. I don’t like Adventureworks, but you could grab a few tables from your system and build a new database, download a pre-built one from SQLskills, or even just create a Hello World type database with one table. Download TeamCity, or use the Visual Studio Online build system. Experiment and learn how to take your database code and programatically deploy changes to another system. Redgate has toolsto help, or you can roll your own.

No matter how you do it, try using a POC the next time you are wondering what the best way to proceed with new technology. You might find the POC makes a new technique more approachable and gives you confidence to tackle deploying the changes in the rapidly changing environments in which we work.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.6MB) podcast or subscribe to the feed at iTunes and LibSyn.

Dexterity Matters

I typo things. I make mistakes in the QOD, checking the wrong box. Even in a browser at times there’s a button or hot area too close to the bookmarks bar and instead of completing an action, like checking my VSTS control panel, I’ll end up on some other website. It’s really annoying, but it’s also an issue that occurs when my muscle memory takes over with my fingers not moving as expected, or moving quicker than the system expects.

In Star Trek or media, we see these flat, touch interfaces similar to smartphones, but there’s never any mention about the missteps or mistakes that could be made by users selecting the wrong item. As we move further from hardware controls to software selections, are we worried about issues? Is this something we run into with our interfaces?

I certainly think so. No matter how often I can do something correctly, I make mistakes. Often simple mistakes, born of impatience. This is especially apparent in Azure and online interfaces where the control, or the browser, might not respond as quickly as I expect. As a result, I might click something multiple times, spawning more actions than I expect. Or I might have an unexpected action occur.

This is one of the great arguments from me for having local software. The interface interactions tend to be more reliable and expected. If nothing else, a local process can allow control of the input buffer, discarding or limiting the multiple clicks that cause issues.

As we move forward, software designers should expect that users will make mistakes and ensure there are ways to cancel or undo accidental mistakes made in an interface. This isn’t always possible, and certainly there are some actions that might not be easily undone, but as we become more dependent on software, we need to consider the potential mistaken actions that users will make and provide methods to correct issues. Especially where data is involved.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.0MB) podcast or subscribe to the feed at iTunes and LibSyn.

Do Less

When I read this blog post, I was transported back to simpler times, when I learned to develop applications on 8088 processors. Back then, to make the system go faster, we learned to do less. It’s certainly true that the less work a system performs, the faster it can complete the limited set of things it must do. Working within C code, most programmers I knew back then tried to optimize algorithms by reducing the amount of work that our CPUs needed to perform.

In T-SQL, we don’t have the low level of control we might have in other languages. In fact, our language has a relatively small number of instructions that we we use to build queries and applications, though that number continues to grow and expand over time. I am grateful for that, but also realize this brings with it additional chances for developers to write code that does more work than necessary.

For much of our work, optimization consists of finding ways to group our operations together. Building a set-based solution takes advantage of the way in which SQL Server most efficiently processes instructions. Far too often there are developers that write inefficient code because they apply techniques they’ve learned from programming applications in some other language, or even from some other relational database.

For a given system, there is a known amount of work that can be done in a period of time. There are also restrictions on the various resources involved (disk, CPU, network, etc). Reducing the load for each of these resources, especially in multi-user systems, can dramatically increase the performance of your system. SQL Server is built to manage concurrency in a variety of ways, but poorly written code is a great way to overwhelm the system.

Do less. Get more performance.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn. 

Building Better Software

One of the ways in which many software development teams have tried to improve is by examining what has worked in other organizations and projects and experimenting with ideas. Across the decades that we have been using computers, there have been quite a few models, the most well known of which is the CMMI Model from Carnegie Mellon University.

The CMMI Model contains lots of good ideas and practices, much like Six Sigma, ISO and other frameworks. However we (as a group) often don’t follow many of the ideas. It seems that so many developers (under pressure from managers) work to just complete tasks, without regards to the well known, valuable techniques of measurement, improvement, and more.

However there isn’t much magic in using any of these frameworks. They all cover similar concepts, which are based on measuring what you do, taking that feedback to alter your process and get better, as well as then working within a structured process that is repeatable. Do more of what works, and do less of what doesn’t. Adjust your way of working regularly.

The problem for most of us is getting a group of people to actually follow a process over and over. It seems that we continually want, at least as a large group, to cowboy code our way through development cycles, without learning from the past. Of course, I meet plenty of people that use an ALM (or DLM) process to build software, and they all espouse the benefits, but they do note that it takes a long time to adopt a new process throughout an organization. A year of investment isn’t considered uncommon.

Perhaps that’s the problem? Can management actually make the investment time to build out better processes? The media hype says they want to, with all the DevOps hype. Plenty of companies, including Redgate Software, are selling tools that companies use to help themselves adopt a process. However I wonder how successful organizations are. Is there enough continuity in the technical staff and management to actually make a change? It took Microsoft years to change. Are you moving in that direction? Is there any chance you will?

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.8MB) podcast or subscribe to the feed at iTunes and LibSyn.

Learning from Microsoft

I know that many data professionals out there are skeptical of the cloud. Certainly there are problems with reliability, security, and more that prevent many of you from migrating to some online database. However that doesn’t mean that everything being done in platforms like the Azure SQL Database is a bad idea.

One of the interesting things that has occurred at Microsoft is a completely new engineering effort for SQL Server. It seems there is one SQL Server development team that works on the Azure SQL Database. This is (I guess) the main branch in version control, and what we see with SQL Server 2016 is a subset of the code being built for the on-premises, boxed product.

Certainly there are some features that exist in the box that aren’t in Azure, but relatively few of them. Those features, and many new ones, have to be included in the version control system, but they can’t be activated. Microsoft must be making extensive use of feature flags, allowing their developers to build and test features in the Azure cloud, but not make them available to customers, or potentially even other parts of Microsoft.

If you really think about the effort to build a platform like Azure SQL Database and allow it to be a development environment as well as a production environment, that’s an amazing engineering effort. From the outside, I’m amazed at how far the SQL Server team has come in the last few years. I’ve seen a number of features released in the cloud first, and then appearing in the SQL Server 2016 CTPs.

I think there are some amazing lessons here for software development. Certainly I think many of us could learn from how feature flags and multiple versions of software can co-exist in systems. That would certainly help many of us deploy new software in a way that minimizes interruptions to customers and clients. Or manage those multiple clients with different requirements from the same codebase.

I hope that Microsoft will continue to evolve, as well as share their knowledge with the world. There are other companies that seem to be accomplishing amazing things at scale, Netflix, Amazon, Google, and more. Some of them share their techniques, but I truly hope that the methods they all use for building software become spread throughout the industry and help all of us build better, more secure applications.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.3MB) podcast or subscribe to the feed at iTunes and LibSyn.

The Dangers of Algorithms

I ran across an interesting piece in Wired recently that talks about algorithms and the power they hold in the modern world. I don’t think it’s a great piece, and it starts to mix code, data, algorithms and other aspects of modern software in a way that seems disjointed to someone that works on software. I think there is come confusion about how data and software differ.

I don’t deny that there are potential negative impacts of the ways in which we process data. Certainly we have bugs in software that cause issues, but we also end up with implementations of specifications that may not quite match up with the expectations of our clients. Even in simple applications, if I’m not constantly interacting with a client in a detailed way, I find that the software can operate in a subtlety different manner than others expect. Of course, there are certainly people that deliberately program systems to intentionally help, or hurt, others, but I think we have just as many issues from small communication mistakes. That’s one reason why DevOps, CI/CD, and frequent releases are valuable. We get feedback quicker on the implementaiton of the design.

Does it make sense to disclose algorithms? Make source code available for examination? I definitely think there are intellectual property issues to consider, but when there are legal challenges or arbitration, I would expect that the actual code of a system can be examined. We definitely need to ensure that code examined in these cases is not part of the public record. However there needs to be open examination in such a way that the groups involved, and perhaps even other interested parties, have confidence that the algorithm performs as expected.

This becomes easier in databases, since our code is often easily accessible, but more complex in applications. Many of us realize that subtle changes in application code can cause issues for data, which is part of the reason we like declared referential integrity in our databases. That way we’re not affected if an application makes a mistake in enforcing our data integrity rules.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.7MB) podcast or subscribe to the feed at iTunes and LibSyn. 

Forcing Strong Passwords

It seems that every week we have some sort of data breach that results in emails and passwords for users being released. That alone is a good reason why you should not have passwords shared across sites or systems. I can’t stress this enough, and if you do this, please go get a password manager today and stop. Really, I mean it.

However is it just the users that create poor passwords? In many cases, sure. However as the people that develop software, we can help. In fact, we should. Security should be on the mind of everyone that writes software.

I ran across a piece on The Hacker News that says far too many sites don’t require users to choose strong passwords. I think this is changing, but far, far too many sites don’t check a user’s password for strength, or reuse. Some even force users to choose less than ideal passwords. I’ve run into plenty of sites that don’t accept the characters in my strong passwords for some reason. Often the length or addition of any special characters if prohibited.

I’ve said it before, but I’ll note it here. Most of us aren’t competent at writing authentication routines and should just use one that’s already written and been vetted. More importantly, we should be sure the code, application and database, is modular enough and can be replaced if we find that our application needs stronger security. Because it will. We are constantly racing Moore’s law to implement better security.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 1.9MB) podcast or subscribe to the feed at iTunes and LibSyn.

Reasonable Timeframes

Many of us hear about problems with various systems on a regular basis. We report them in Database Weekly far too often, and I’m sure a few of you have been informed (or found) bugs, nagging issues, even vulnerabilities in your systems.

What’s a reasonable time to fix a system after an audit?

It’s a simple question, but it’s not very simple to answer. After all, most of us don’t have a lot of slack development time built in to fix issues. Unless the issue is a broken application that doesn’t work, the items disclosed in an audit need to scheduled in among other work. After all, most of the time the audit finds something that no one is aware of, or no one has wanted to fix. This is work that no one really planned on completing.

I ran across an interesting piece about the Employment Department for the state of Oregon hasn’t fixed a number of issues after an audit last year. While some strides have been made, there are still outstanding issues, the sum total of which it is estimated will take a decade to complete. That’s a long time, but in large systems, especially ones where the entire application cannot be rewritten because of resources, it’s not unusual. I’ve worked in a few places where we had large scale systems that we knew had issues, but we couldn’t easily re-design and implement fixes in any reasonable length of time. Often this was because of downstream dependencies, but certainly culture and management hadn’t made change a priority.

I sympathize with those people dependent on mainframe systems. The power and reach of those systems, the poor documentation, not to mention the complex training required to change clients’ habits is huge. I would hope that the government groups using these large scale systems would work together to jointly proceed on development, with politicians also involved to help standardize the requirements across state lines (or countries’ borders) and simplify the software needed.

However, no one ever makes software simpler, especially when it’s being designed.

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.4MB) podcast or subscribe to the feed at iTunes and LibSyn.

Queues in Databases

I ran across this post, The Database as a Queue Anti-Pattern, and it struck a chord. I both agree and disagree with the premise, and certainly see this as another case of the “it depends.” However, I am curious what others think.

I had just written about this for T-SQL Tuesday #74, talking about our use of the database for queuing emails as a part of the site. It was a high volume system that worked for us. We were, of course, looking at SQL Server as a hammer, and in 2002/2003, there weren’t many messaging systems available at a reasonable cost. These days it seems that Amazon SQS or Azure Service Bus are low cost ways to move messages around, but are they better than a database?

In our case, cost mattered, knowledge and skill mattered, though certainly we could have learned how to work with a messaging system. However we also had the need to feedback potential status and progress information. We also wanted the ability to re-point emails to another process if one failed. For example, if SenderB fails after having retrieved an email, we can update the table and have SenderA pick up the row(s) that haven’t been processed.

I do believe that messaging is an extremely powerful architecture technique for for distributing work as well as scaling to high workloads. I’d like to see more messaging implemented in applications, even those that aren’t Twitter scale. After all, if I exceed the power of a 4 core machine for my small application, it’s much easier for me to add another 4 core machine that can deal with messages than upgrade to an 8 core machine and just delay the scale problem to another day.

I’m curious what the rest of you think. Is the database a good platform for a queue?

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.3MB) podcast or subscribe to the feed at iTunes and LibSyn.