Test if a File Exists with Powershell

Another post for me that is simple and hopefully serves as an example for people trying to get blogging as#SQLNewBloggers.

Awhile back I wrote a PowerShell script (PoSh) to download the SQL Saturday XML files used for the Guidebook application. These contain a bunch of the information that I wanted to collect, transform, and keep around.

However the script wasn’t great. It basically looped a set number of times and re-downloaded the files. Not the more efficient solution, especially if I want this to run regularly.

One of the enhancements I wanted to make was check if the file exists, and if not, then download it. However, if it does exist, then I’ll skip the file. I know this means I don’t get updated files if schedules change, which is possible, but in that case, I can just delete the file from my desktop and go from there.

I made a quick search, and found a few links to the Test-Path cmdlet. Essentially you give this a file path and it returns true or false. Almost exactly what I need. This is what I added to my code:

if (Test-Path $DestinationFile) {

#do something


However I want to take action if the file doesn’t exist. In most languages, I’d add a ! in front to signify "not". However that doesn’t work in PoSh, just like > doesn’t mean greater than.

Another search showed me the -NOT operator. That’s what I need, but I can’t do this:

if (-NOT Test-Path $DestinationFile) {

#do something


Instead, I need to have a single expression for -NOT, which means more parenthesis. Not a big deal. I used this code:

if (-NOT (Test-Path $DestinationFile)) {

#do something


That worked great and now I only download the files I need. If I want to re-download (for newer events), I just delete those files and re-run the script.


This post came about when I started working on the script. It actually took longer to write this than find the solution, add it to code, and test it. That process was about 5 minutes.

This post took about 10 minutes to write. I also had a second post based on similar modifications to the script, so I’m did that in another 5 minutes.


A few of the links I used:

Just Say No

Just say no was a tagline from my youth. As I recall, it didn’t work well then, but I hope it will work better for a different audience and in a different context today. I was reading Brent Ozar’s note about one of the toughest things for consultants to do: saying no. While I certainly think this isn’t a problem for some people that can’t find work, I do think this is an issue, and one that isn’t just a problem for consultants.

I have worked with many people across my career, and one of the things that most of them have struggled with is saying no. Maybe it’s fear over being scolded (or worse) by a manager. Maybe it’s the simple desire to please others. Perhaps it’s because we simple are eternal optimists, and we think we can handle everything thrown at us. No matter what your reason, many people find the inability to say no plagues them throughout their careers.

I’ve learned to say no across the last decade. I probably say no to more things than I agree to accomplish. There are times that I worry about how much I decline to tackle things at work. However I have learned to deliver what I agree to, and deliver it well. I’ve learned that to do that, I can’t tackle every project, idea, or assignment. There are times I have to push back and refuse (politely) to do something.

It’s a luxury for sure. I have been successful in my career, and I’m not in a position where the loss of my job would wreck my life. However I also realize time is the most valuable resource I have at work. Time is also the most valuable resource I have in life, and I have to learn to balance my use of that resource to continue to be successful. Hopefully you learn that as well.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.1MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

The Biggest Data Breech (For Now)

I keep reading the words “the largest data breach in history” in a variety of stories. In fact, given the tremendous growth of data acquisition, I’m guessing that this headline will continue to repeat itself over and over. I think I’m getting to the point where I’d rather just see a story say that xxx million customers were affected. At least then I’d be able to easily put some scale to the loss of data.

What’s interesting in this case involving JP Morgan is there are indictments being handed down, to at least two men that somehow participated in hacks that copied over 100million people’s data. JPMorgan admits 76 million households and 7 million small businesses were compromised, which isn’t 100, but perhaps there’s something I’m missing. However the data wasn’t just sold, but rather hackers used the information to market stocks to the individuals compromised. That’s an interesting level of sophistication, and a scary one.

Can you start to imagine criminals using the information intelligently to not directly sell the data but to make a secondary use of the information. Perhaps they will enagage social engineering by bundling the information with other data to perform some other attack on individuals? It’s entirely possible that we will see more sophisticated uses in the future as criminals work to evade or avoid the fraud detection systems that have been put in place.

I have no doubt that bigger data breaches are coming. Perhaps we could reduce the impact and frequency with better security frameworks and development practices, but I’m not sure that any company out there will place a high priority on security over ease of access and speed of development. I do continue to hope that market forces will drive companies to build better detection and protection mechanisms, and our vendors will build better security mechanisms into all platforms.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.4MB) podcast or subscribe to the feed at iTunes and LibSyn.

Visual Studio Subscriptions

Many of us that work with SQL Server do so exclusively through SQL Server Management Studio (SSMS). I find so many people really do the majority of their jobs with SSMS, Outlook, and a web browser. Even back in 2003 when I was a full time DBA, I probably spent the majority of my time in those three applications.
However I also see more and more people using Visual Studio and other tools to accomplish their jobs. The growth of new tools, like Powershell, the expansion of our work into BI areas, and more mean that more and more people are using tools besides SSMS to work with SQL Server data.
This past week there was an announcement that MSDN subscriptions were changing. At most of my jobs, I’ve had an MSDN subscription available to me. In fact, some of you might remember the large binders of CDs (and later DVDs) that arrived on a regular basis and contained copies of all Microsoft software. However many of you out there haven’t had MSDN available to you, or you’ve struggled to justify the yearly $1000+ cost, but you do want to work on your careers and practice with Microsoft software.
At first I saw the yearly cost of MSDN at $799, which is a pretty large investment. However as I looked to the side, I saw a monthly subscription, no large commitment, available for $45. That’s not an extremely low cost for much of the world, but it’s very reasonable in the US. It’s also a great way to build a setup that allows you to work with a variety of Microsoft technologies at an affordable cost. What’s more, you can stop paying at any time. Or start again at any time.
I know that it can be a struggle to invest in your own career, probably more difficult to find time than money. However this is a good way to get access to the various development and server tools for a period of time if you want to tackle a project or force yourself to learn a new skill.
I’m glad that Microsoft has moved to a subscription model for MSDN. I expect to see this subscription growing as small companies use a small investment that scales linearly with new hires to provide their employees with tools. I can only hope that many other vendors adopt this same model and allow us to rent our tools, and upgrade, for a very reasonable cost. I just hope they all let us backup and save our settings in case we interrupt our subscription for a period of time.
Steve Jones

Technical Debt

I was speaking with one of the development teams at Redgate earlier this year. They were working on a product, and had planned out a few sprints worth of work. Each sprint, called a train, was a couple weeks long, with specific goals and ideas to be implemented. That was all good, but I noticed that there was a sprint in the middle of the list that was devoted to technical debt.

Technical debt is a strange term. It’s one that many managers don’t understand well, often because the code may work fine. I ran across an interesting piece that looks at what the concept means, with what I think is a good explanation. We get technical debt when we sacrifice maintainability to meet another requirement. The piece also looks at the accumulation of the debt and why it becomes problematic later. Certainly the more debt that piles up, the mode difficult it can be to change code. Since we are almost always going to go back and maintain code, this becomes a problem.

I think the ideas given to keep technical debt under control are good ones. We should make an effort to clean code as we can, though not make it such a priority that we end up causing more work with constant refactoring. We do need to get work done. However the suggestions given require a good amount of discipline and buy in from management, and I’m glad Redgate tries to keep debt under control. I think our developers like the debt trains as well.

I thought the idea was pretty cool until I was looking for a feature to be completed and the technical debt train was running that week. I thought about complaining, but I decided to have patience and wait a week. After all, if the debt isn’t kept under control, I might be waiting much longer than a week for some fix or feature next year.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.4MB) podcast or subscribe to the feed at iTunes and LibSyn.

Branding Yourself for a Dream Job–Powerpoint and Questions

I had a few people ask about the deck, and I’ve uploaded it here: Branding Yourself for a Dream Job.

I also had a couple good questions at the end after people had left, so I thought I’d put them here.

What about a picture on my LinkedIn/resume?

I have had a few people ask this in the past, and I forgot to cover this in the talk. My only advice is that if you want to put one up, use a professional picture. Have someone take one.

Other than that, it’s a hit and miss prospect. We all have prejudices, and hiring managers/HR people are no different. If you’re a women, or old, or a minority, you might have some people exclude you based on your picture.

Personally I don’t see much benefit to adding a picture. Keep that information back until you need to disclose it.

Does it matter how many blogs I have?

No. However you will likely want to maintain them, so don’t tackle too much. I would suggest no more than 2, one for your career and one for something else.

Python and Tweepy

One of the projects that’s been on my list lately is to programmatically access Twitter for a few ideas I want to play with. Since I’ve been trying to learn some Python, I thought I would take a look using Python to update status and read status.

A quick Google search showed me lots of Python clients, but Tweepy caught my eye for some reason. I’m not sure why, but I ended up popping a command line open and downloading the library.

From there, I saw a short tutorial over at Python Central. I started by creating an app at Twitter for myself, which was very simple. Once that was done, I had a set of consumer tokens (key and secret), that I could use. Another click got me to the access key and secret. Note, the easy way to do this is over at dev.twitter.com.

My first attempt was using this sample code.

import tweepy

consumer_key = “ABCDE”
consumer_secret = “12345”
access_token = ‘asdfasdf’
access_token_secret = ‘98765’

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

api = tweepy.API(auth)

public_tweets = api.user_timeline()
for tweet in public_tweets:

This gives me a list of my tweets. At least, the last 20.

2015-11-06 13_30_24-SDTIG - [C__Users_Steve_OneDrive_Documents_Python_SDTIG] - ..._Last_10_twitter_l

That’s progress.

I then went to make an update by using this:

api.update_status('Hello, Pyton')

However that returned an error:

2015-11-06 13_50_53-Cortana

Hmmm, I double checked the API, and this should work, but I’m guessing there’s another issue. I searched, and found there’s a bug. However, I should be using named parameters, so no big deal. Change the code.

api.update_status(status='Hello, Pyton')

Now it works.

2015-11-06 13_53_06-Steve Jones (@way0utwest) _ Twitter

This is the first step for me to look at building an app that might send some tweets on my behalf, perhaps with the data stored somewhere, like, I don’t know, maybe a database?

Shrinking the Budget

Most of us rarely have to build or manage budgets in our organizations, but almost all of us are affected by the budget process. It’s tempting to ignore budgets and just do your job, but sooner or later you might find yourself arguing for more funding for a project, training, or even hardware.

I ran across a short piece on 10 ways to shrink your IT budget and found it a bit scary. I would bet that many of you have had conversations, or been affected by decisions, that follow some of the advice in the piece. The push for open source, virtualization, hosted (or cloud) migrations, or more can cause stress and anxiety when your job is making sure the servers run without complaints from end users.

I don’t expect that all of these would be followed in a specific organization, but some of these might be proposed to you. What I would recommend is you understand the reasons why, or why not, you might adopt any of these ideas. Certainly the time lost from retraining people or rewriting code can overwhelm the savings for many years, but even seemingly smaller changes, like changing priorities can affect the way your clients and customers view IT. It seems that sometimes budgets get changed to provide a short term view that technology spending is being managed efficiently, only to find out later we will remove all our savings by undoing a poor decision.

I’d suggest that you approach budget issues with transparency and honesty. Take a hard look at your costs and determine what items are really needed, and what items aren’t. However, I’d also urge you to carefully consider whether it’s really valuable to save money by not taking care of your staff. While labor is an expensive part of your IT cost, good staff are worth much more than they cost, often by an order of magnitude.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 2.0MB) podcast or subscribe to the feed at iTunes and LibSyn.

Viewing Extended Properties for Information

Another post for me that is simple and hopefully serves as an example for people trying to get blogging as#SQLNewBloggers.

I’ve been working a little with extended properties, adding and updating them for various objects. However in addition to altering properties, viewing the properties on an object is often necessary. This post will look at how we get the properties in a few different ways.

The easiest way to see extended properties is to look at the properties of an object in the SSMS Object Explorer. For example, I can right click on a table in OE.

2015-11-02 20_30_55-

Once I click Properties, I get a dialog with a lot of items on the left. The bottom one is for Extended Properties, with a simple add/edit/delete grid. Here I can see the property(ies) I’ve added.

2015-11-02 20_31_07-Table Properties - SalesHeader_Staging

However this is cumbersome for me. I’d much rather find a way to query the information, which is what I need to do with an application of some sort. I’d think sp_help would work, but it doesn’t. If I run this, I get the following result sets:

  • header with owner, type, and creation date.
  • column list with meta data
  • identity property information.
  • RowGuid column information
  • filegroup storage location.
  • Messages with index, constraint, FK, and schemabinding relations.

Not very helpful in this case.

I do know that extended property information is in sys.extended_properties. I can query this view, which gives me some information, but I need to join this with sys.objects for easy to understand information.

2015-11-02 20_38_42-SQLQuery13.sql - aristotle.RaiseCodeQuality (ARISTOTLE_Steve (69))_ - Microsoft

This works, and this is one of the ways in which I do query properties in various tSQLt tests.

There is one other way I’ve seen to query extended properties. When perusing the BOL page for sp_updateextendedproperty, I found sys.fn_listextendedpropery. This is a DMF, a function, that you can use to query for property values. Since it’s a TVF function, I need to use it in a query as a functional object.

2015-11-02 20_42_27-SQLQuery13.sql - aristotle.RaiseCodeQuality (ARISTOTLE_Steve (69))_ - Microsoft

There are lots of parameters in this function. However you can guess what they are after working with the other extended property procedures. In fact, the first time I started this post, I was disconnected and had to experiment with the function, adding parameters until it ran without an error.

The first parameter is the name of the property. This can be NULL, in which case you’ll get all the properties that exist.

2015-11-02 20_44_48-SQLQuery13.sql - aristotle.RaiseCodeQuality (ARISTOTLE_Steve (69))_ - Microsoft

The rest of the properties correspond to the level 0, 1, 2 types and names that you are using to filter the results. This is actually a good technique to use with this function, and I’ll be using this more in the future.


This post followed on from the previous ones. In this case, I started this disconnected, using the knowledge I had to write the basics with SSMS and the system table. That took about 20 minutes to document and then I spent 5 minutes experimenting with the function, whose name I had on an open browser tab. Once I worked through that, I spent another 5 minutes writing.

Thirty minutes to a post. You can do this.


A few items from BOL:

sp_help – https://msdn.microsoft.com/en-us/library/ms187335.aspx

sys.extended_properties – https://msdn.microsoft.com/en-us/library/ms177541.aspx

sys.fn_listextendedproperty – https://msdn.microsoft.com/en-us/library/ms179853.aspx

Relational By Default

I’ve read a few pieces from Vladimir Khorikov and I tend to like the thoughtful way in which he approaches building software. When I saw a link to his SQL v NoSQL databases, I was intrigued to see what his thoughts would be. It’s a good read, though with relatively few comments or debates posted at the time I read the article. I was hoping for more and I encourage you to add your thoughts.

The main contentions in the piece are that any of the simpler development techniques that we can use with NoSQL databases don’t remove the need for implementing data quality or management features. They just require those features to be implemented by the developer. Specifically the need to deal with multiple schema versions over time, or the requirement we enforce parent child relationships.

Perhaps that’s not too bad. After all, if we evolve schemas and the code can easily deal with the changes, that’s good. However I think the point about having relational storage when you can, and adding in other stores makes sense. This is what I really believe as well, since we often need to query and combine data, which is something relational stores do very well and very efficiently.

Ultimately I’m sure we will continue to see arguments about relational and NoSQL models being better or worse for particular problems. I actually welcome the arguments  because there are certainly domains of problems where one system works better than the other, and it’s good to develop some clarity about those particular problems. However there are also likely many situations where multiple platforms can work, and in those cases, we should use what works well for our team.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 1.9MB) podcast or subscribe to the feed at iTunes and LibSyn.