Adding the ReadyRoll Extension in VSTS

As I’m doing a little development with ReadyRoll, I’m documenting some of the things I learn about the product.

A few weeks ago I needed to do a demo with ReadyRoll for Redgate Software. I built a new project, and wanted to perform a build in Visual Studio Team Services (VSTS). However, I realized that I needed to install the extensions for my project to work. This is pretty easy, but worth a short post.

If I create a new VSTS project, I get a blank, empty project page. 2016-07-15 11_30_27-Highlight

If I navigate to the Release page, of course, I have nothing there.

2016-07-15 11_38_51-Highlight

And when I look to add something to a release definition, I have a fairly small list of tasks from which I can choose. These are in alphabetical order, and there’s no “r” tasks.

2016-07-15 11_49_57-Highlight

I do need a build setup, but once that’s done, I still have the same list of three items. I want to add the ReadyRoll extension. To do that, I’ll click the shopping bag in the upper right of the screen. Once I do that, you’ll see I can browser or manage extensions.

2016-07-15 11_41_15-Highlight

I’ll choose Browse, which opens the marketplace. There are extensions for many, many developer tasks, and I’d encourage you to look through the list.

2016-07-15 11_41_45-Highlight

In my case, I’m looking for ReadyRoll, so I’ll enter Redgate to search.

2016-07-15 11_41_55-Highlight

There are three results, the first of which is ReadyRoll.

2016-07-15 11_42_02-Highlight

I can click on the box, and I’ll get more details on the extension.

2016-07-15 11_42_10-Highlight

The extension is free, but I do need ReadyRoll on my desktop to build a ReadyRoll project in Visual Studio. As the page notes, you can download a 28 day trial to play with the product if you like.

If I click “Install”, I’ll get a dialog asking me to confirm the installation to this account. Note there is a drop down if you have access to multiple accounts. The permissions needed are also listed.

2016-07-15 11_47_30-Highlight

Once this is installed, I get a confirmation:

2016-07-15 11_48_14-Highlight

And now when I go to add a task, I see ReadyRoll listed.

2016-07-15 11_49_11-Highlight

I click “Add” and then can configure the task.

2016-07-15 11_49_25-Highlight

That’s how easy it is to add ReadyRoll, or really any extension to your VSTS account.

MeasureUp 2016

I’m heading down to Austin next weekend for MeasureUp 2016. This is a Microsoft stack conference that looks to help people build custom business software.  There are over 20 sessions, most of them devoted to .NET topics. I think I’m more excited to be in some nice company and have the chance to sit in on some interesting talks.

I’ve got the one database talk, Bringing DevOps to the Database. I’ll be looking at some of the ways in which we can change database development to be more agile, to fit in more with the way that most application developers build code.

This builds on work I’ve done in the past, as well as a lot of work that Redgate has done across the last 4-5 years.

If you’re in the Austin area, and you develop software, maybe you want to come down on Saturday for the event. It’s low cost, and the chance to meet with other software developers (and get out of the heat).

Register today for MeasureUp 2016 and come shake my hand.

Starting ReadyRoll with an Existing DB

I’ve been working with ReadyRoll quite a bit the last few months, and wanted to start putting out some information. I’m looking to do a longer series, but here’s a quick post on getting started.

When you install ReadyRoll, you get a new project type under SQL Server.

2016-06-14 12_27_14-Photos

This is a database project that will track all changes to your database as a series of migration scripts. Let’s get started. I’ll choose this and give it a name.

Once I do this, I get the ReadyRoll window with some information. What I really want to do here is point this to my existing database. Right now it’s looking at a LocalDB instance.

2016-06-14 12_28_07-Photos

I’ll click the configure link and get the Connection String dialog. This is the database that mirrors one of the SSC databases, with lots of objects in it.

2016-06-14 12_29_38-Photos

Once I’ve done this, the project will use this database. The Shadow database will also be on this instance.

I need to make one other change and set the correct version in the Project Settings.

2016-06-14 12_48_29-Photos

Now, back to the ReadyRoll window. I’ve configured the DB connection, so I can ignore that. What I want to do is import my database, getting a baseline script.

2016-06-14 12_49_32-Movies & TV

I click that and ReadyRoll creates a Shadow database (since one doesn’t exist) and begins to import objects.

2016-06-14 12_49_39-Photos

Once that’s done, I get a list.

2016-06-14 12_49_48-Photos

And a migration script.

2016-06-14 12_49_58-Movies & TV

This is my baseline, starting script. This contains all the objects that exist in the database at this point. These are also added to the Schema-Model, but I can ignore those. I’m going to work on the database itself.

The ReadyRoll widget notes this was an import, and there’s no need to deploy anything since these aren’t changes, but just the state of the db.

2016-06-14 12_52_09-Movies & TV

I can see this if I do a Visual Studio build. Note the message in the middle: No migrations pending deployment. The changes in script 1 (001_20160614-1249_sjones.sql) are already in the db.

2016-06-14 12_52_50-Photos

Now I can make changes to my database and get new scripts. Add a table in the designer?

2016-06-14 12_54_55-Photos

When I click Update (and Generate Script), I get a new migration script.

2016-06-14 12_55_09-Movies & TV

Note that I just generated this script. I’ll write more about this process later, but for now, I’ll click Deploy Project to execute this against my database. When I do that, VS does a build, and one migration is executed.

2016-06-14 12_56_06-Photos

Add a procedure?

2016-06-14 12_57_13-Photos

Generate a new migration script.

2016-06-14 12_57_35-Movies & TV

And so it goes. I can work with my database in VS and get new scripts. I can also do an import if someone else makes changes to the database from their own machine with VS, SSMS, SQLCMD, isql, etc. The import will generate another migration script that gets added.

This is a really basic look at ReadyRoll, but it will get you started. I’ll tackle more topics and different ways of working with database development in another series of posts.

Running Multiple Threads Quickly

Recently I was looking to generate a quick workload against an instance. I wanted to add to the amount of work my system was doing by executing a number of queries simultaneously. There are a variety of ways to do this, but I just wanted a quick set of queries. I’ll show a couple ways to do this and then look at a few other tools in later posts.

The Cumbersome Way

I can open queries in two windows in SSMS. Note, each of these will execute 50 times.

2016-05-20 14_17_34-SQLQuery1.sql - (sa (56))_ - Microsoft SQL Server

Now I have two windows, and I can click execute in one, then click to the other, and click execute again. That’s easy. When I do this, I’ll have two threads, each running a query 50 times.

A Better Way

A better way is to use a SQLCMD call with my query in it. In this case, I’ll create a notepad file and add multiple SQLCMD calls in it.

2016-05-20 14_29_03-SalesDemo-2015-12-01-1745-export-i-fgod4b6h - VMware Workstation

The key here is the “start” at the beginning of the line. This will spawn a new thread with the program being called in it. In this case, I’ll get 5 windows very quickly, each running my query. My query is in another file:

2016-05-20 14_28_19-SalesDemo-2015-12-01-1745-export-i-fgod4b6h - VMware Workstation

If each query is set to run multiple times, I’ll have a simple load generated. In my case, I’ll run the .CMD file from the command line, but I could double click it. When I do, I see this:

2016-05-20 14_30_13-SalesDemo-2015-12-01-1745-export-i-fgod4b6h - VMware Workstation

You can see the window where I started the queries in front. Three of the command windows are in the background, each of them running queries over and over. The output from the query, with all the dashes for spacing between the headers and data, are in each window.

Successful Projects

What techniques or ideas allow you to build a successful software project? There’s been no shortage of writing and studying on this topic across the last few decades. So many people expend energy and thought trying to solve the problems of building software at a high level of quality, in a reasonable timeframe, and within some scope that matches the desires of their customers.
I’d like to think most of us succeed, but study after study shows that most parties are disappointed in some way. Developers are under pressure and can’t/don’t built the level of quality they’d like. Management sees software as taking too long and costing too much. End users see too many bugs, though I think that overall quality has risen in the last decade are more proven frameworks are used and developers have gained more experience in newer and better development techniques.
I ran across a post on a test for successful software projects. It’s the Kevin Test for Successful Projects, and it’s based on a combination of the Joel Test and the Rands Test. The idea is not that you implement all of these test items, or do them perfectly, but that you work towards doing more of them than not. There are explanations of what each item in the test means in the various pieces.
I think these are good guidelines for projects, and they’re simple ones. They certainly can apply to database development. However most people don’t really want to spend time on these tasks. However, I think the mundane nature of these items slows you slightly, but forces you to think more about what you’re doing. The gains in quality and velocity, in my mind, come from avoiding rework, avoiding losing work, and avoiding regressions.
Sometimes I think the problems in software development aren’t that we write poor code, but rather that we rush a complex endeavor, being willing to redo work rather than proceeding in a more engineered fashion.
Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.8MB) podcast or subscribe to the feed at iTunes and LibSyn.

Build a POC

I have a few friends that work sales engineers for various companies. They often build a proof of concept (POC) project for various customers or potential customers, which is a great job. They get to showcase how a particular product or technology might work in a situation. We see this all time from lots of companies, including a great POC from BMW shown at the recent //build/ conference.

Sometimes the POC is more mockup than reality, but often this is a way to actually test some software and determine if you should invest more resources in that particular direction. I’ve used this myself, setting up a small project to test items, and I recommend this for others that might want to try something out.

For example, I like the idea of Continuous Integration (CI) for databases, but implementing CI in one of your production pipelines, on a production database is hard. It’s complex, and it can be distracting, not to mention stressful. Who wants to make a deployment mistake in production?

What I recommend instead is that you take a small database and practice. I don’t like Adventureworks, but you could grab a few tables from your system and build a new database, download a pre-built one from SQLskills, or even just create a Hello World type database with one table. Download TeamCity, or use the Visual Studio Online build system. Experiment and learn how to take your database code and programatically deploy changes to another system. Redgate has toolsto help, or you can roll your own.

No matter how you do it, try using a POC the next time you are wondering what the best way to proceed with new technology. You might find the POC makes a new technique more approachable and gives you confidence to tackle deploying the changes in the rapidly changing environments in which we work.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.6MB) podcast or subscribe to the feed at iTunes and LibSyn.

Dexterity Matters

I typo things. I make mistakes in the QOD, checking the wrong box. Even in a browser at times there’s a button or hot area too close to the bookmarks bar and instead of completing an action, like checking my VSTS control panel, I’ll end up on some other website. It’s really annoying, but it’s also an issue that occurs when my muscle memory takes over with my fingers not moving as expected, or moving quicker than the system expects.

In Star Trek or media, we see these flat, touch interfaces similar to smartphones, but there’s never any mention about the missteps or mistakes that could be made by users selecting the wrong item. As we move further from hardware controls to software selections, are we worried about issues? Is this something we run into with our interfaces?

I certainly think so. No matter how often I can do something correctly, I make mistakes. Often simple mistakes, born of impatience. This is especially apparent in Azure and online interfaces where the control, or the browser, might not respond as quickly as I expect. As a result, I might click something multiple times, spawning more actions than I expect. Or I might have an unexpected action occur.

This is one of the great arguments from me for having local software. The interface interactions tend to be more reliable and expected. If nothing else, a local process can allow control of the input buffer, discarding or limiting the multiple clicks that cause issues.

As we move forward, software designers should expect that users will make mistakes and ensure there are ways to cancel or undo accidental mistakes made in an interface. This isn’t always possible, and certainly there are some actions that might not be easily undone, but as we become more dependent on software, we need to consider the potential mistaken actions that users will make and provide methods to correct issues. Especially where data is involved.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.0MB) podcast or subscribe to the feed at iTunes and LibSyn.

Do Less

When I read this blog post, I was transported back to simpler times, when I learned to develop applications on 8088 processors. Back then, to make the system go faster, we learned to do less. It’s certainly true that the less work a system performs, the faster it can complete the limited set of things it must do. Working within C code, most programmers I knew back then tried to optimize algorithms by reducing the amount of work that our CPUs needed to perform.

In T-SQL, we don’t have the low level of control we might have in other languages. In fact, our language has a relatively small number of instructions that we we use to build queries and applications, though that number continues to grow and expand over time. I am grateful for that, but also realize this brings with it additional chances for developers to write code that does more work than necessary.

For much of our work, optimization consists of finding ways to group our operations together. Building a set-based solution takes advantage of the way in which SQL Server most efficiently processes instructions. Far too often there are developers that write inefficient code because they apply techniques they’ve learned from programming applications in some other language, or even from some other relational database.

For a given system, there is a known amount of work that can be done in a period of time. There are also restrictions on the various resources involved (disk, CPU, network, etc). Reducing the load for each of these resources, especially in multi-user systems, can dramatically increase the performance of your system. SQL Server is built to manage concurrency in a variety of ways, but poorly written code is a great way to overwhelm the system.

Do less. Get more performance.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn. 

Building Better Software

One of the ways in which many software development teams have tried to improve is by examining what has worked in other organizations and projects and experimenting with ideas. Across the decades that we have been using computers, there have been quite a few models, the most well known of which is the CMMI Model from Carnegie Mellon University.

The CMMI Model contains lots of good ideas and practices, much like Six Sigma, ISO and other frameworks. However we (as a group) often don’t follow many of the ideas. It seems that so many developers (under pressure from managers) work to just complete tasks, without regards to the well known, valuable techniques of measurement, improvement, and more.

However there isn’t much magic in using any of these frameworks. They all cover similar concepts, which are based on measuring what you do, taking that feedback to alter your process and get better, as well as then working within a structured process that is repeatable. Do more of what works, and do less of what doesn’t. Adjust your way of working regularly.

The problem for most of us is getting a group of people to actually follow a process over and over. It seems that we continually want, at least as a large group, to cowboy code our way through development cycles, without learning from the past. Of course, I meet plenty of people that use an ALM (or DLM) process to build software, and they all espouse the benefits, but they do note that it takes a long time to adopt a new process throughout an organization. A year of investment isn’t considered uncommon.

Perhaps that’s the problem? Can management actually make the investment time to build out better processes? The media hype says they want to, with all the DevOps hype. Plenty of companies, including Redgate Software, are selling tools that companies use to help themselves adopt a process. However I wonder how successful organizations are. Is there enough continuity in the technical staff and management to actually make a change? It took Microsoft years to change. Are you moving in that direction? Is there any chance you will?

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.8MB) podcast or subscribe to the feed at iTunes and LibSyn.

Learning from Microsoft

I know that many data professionals out there are skeptical of the cloud. Certainly there are problems with reliability, security, and more that prevent many of you from migrating to some online database. However that doesn’t mean that everything being done in platforms like the Azure SQL Database is a bad idea.

One of the interesting things that has occurred at Microsoft is a completely new engineering effort for SQL Server. It seems there is one SQL Server development team that works on the Azure SQL Database. This is (I guess) the main branch in version control, and what we see with SQL Server 2016 is a subset of the code being built for the on-premises, boxed product.

Certainly there are some features that exist in the box that aren’t in Azure, but relatively few of them. Those features, and many new ones, have to be included in the version control system, but they can’t be activated. Microsoft must be making extensive use of feature flags, allowing their developers to build and test features in the Azure cloud, but not make them available to customers, or potentially even other parts of Microsoft.

If you really think about the effort to build a platform like Azure SQL Database and allow it to be a development environment as well as a production environment, that’s an amazing engineering effort. From the outside, I’m amazed at how far the SQL Server team has come in the last few years. I’ve seen a number of features released in the cloud first, and then appearing in the SQL Server 2016 CTPs.

I think there are some amazing lessons here for software development. Certainly I think many of us could learn from how feature flags and multiple versions of software can co-exist in systems. That would certainly help many of us deploy new software in a way that minimizes interruptions to customers and clients. Or manage those multiple clients with different requirements from the same codebase.

I hope that Microsoft will continue to evolve, as well as share their knowledge with the world. There are other companies that seem to be accomplishing amazing things at scale, Netflix, Amazon, Google, and more. Some of them share their techniques, but I truly hope that the methods they all use for building software become spread throughout the industry and help all of us build better, more secure applications.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.3MB) podcast or subscribe to the feed at iTunes and LibSyn.