Separation of Duty

 In the Vietnam war we found people gathering data that had to support the analysis or decisions of the leaders. That’s a fundamental problem. Arguably this led to the extension of the war, and more lives lost, without more decisive actions being taken.

We should not have the people or systems designed to collect data that supports a particular position. We want data that is accurate and complete, which can then be used for analysis. We ought to have people gathering data separate, both in terms of hierarchy and process, from the people that analyze and make decisions. We know that keeping them in the same area leads to data that supports what the boss wants, or what the bonus warrants, rather than what the data can show.

Have you heard of the Franklin Gambit? I was reminded of that recently while reading Obliquity, a book that examines how we pursue goals and achieve success. Franklin’s Gambit says that we often use data and explanations to justify a decision we’ve already reached, rather than actually prove that some choice is the best one. I’m sure most of you feel you don’t live your live this way, but as I’ve thought about some of my decisions, I think this isn’t necessarily true.
It’s a natural tendency of humans to do justify their actions, regardless of the data, and I suspect many of us suffer from it. I think it also can extend to the data analysis and interpretation that many of us perform at work. We’d like to think that our reports and even data gathering are objective ways to examine a problem, but it can be easy to influence the data, especially in the way that data is gathered if we are not careful.
We want to separate the data gathering from the analysis. It might not ensure that the data analyzed is as complete and accurate as possible, but it helps prevent one side from influencing the other.

This is Big Data

I’m sure some people have bigger data, but Facebook has some serious daily growth. The highlight? 500TB a day.

A day.

That’s serious data. I’ve never had a database that was a TB, must less growth by over a TB a day.

To be fair, lots of the Facebook data is unstructured, stored in Hadoop or NoSQL structures for images/video, but there’s a ton of database-like status updates, links, comments, etc. that have to be managed.

Rather impressive, but not something I want to administer.


Inertia can be hard to overcome

This editorial was originally published on August 22, 2006.

My son asked me about physics the other day and what it meant. He’s been on a bit of a science kick, looking to understand and explain his world. I can understand that because most of us are curious about how things work in our world, not necessarily in a scientific way, but just in general.

I somewhat butchered the definition because I was caught up slightly in the actual concepts, but one of the things we talked about was inertia. Maybe because I’d seen this article on overcoming institutional inertia the same day.

We all have things at work that we do a certain way. Maybe we learned it that way, maybe we’re more comfortable, or maybe someone yelled at us daily until we did it that way. Regardless, fighting the way people work is a difficult thing to undertake. The article gives a few examples of process changes and how they were effectively moved forward and how the issues were resolved at a high level. It’s common sense if you’re wondering, use diplomacy and take your time.

I’ve seen changes occur from both sides of the fence, from someone trying to implement change and from someone who was told changes were occurring. I was never completely comfortable in either situation mostly because with change there’s usually conflict. Someone doesn’t want to change.

As DBAs we often try to control our world and ensure a high degree of stability, often at the cost of slowing or severely impeding changes in our systems. While I think it’s good to aim for a stable, reliable environment, I don’t think this should necessarily impede change.

Understanding the business needs and goals is critical both for DBAs in protecting their systems and developers in building new applications. In either case, it also means that the needs of your organization need to dictate how quickly things change.

Not an employee’s personal feelings.

Changing the Past

The National Archives

A few years ago my accountant caught a mistake in one of our tax returns. This was during the next year’s filing and so we had to amend to previous return, filing against almost two years later to correct an issue. Fortunately the error was in our favor and we ended up receiving a refund.

In most businesses you typically do not change data that was completed in the past unless there is a severe error. For some types of data, such as audit data, you never want to change it. And in many cases, even if there were some change, because of the way that you have operated the business based on those past values, you might decide not to change things.

However as more and more data is accumulated, and used in business decisions, legal proceedings, etc, it is likely that more data professionals will be faced with the questions about when it is appropriate to alter something. With that in mind, this Friday’s poll is:

When would you change archived data?

Are there circumstances that you are aware of where it is valid to change data that might be archived? Is it only to make corrections that were recorded incorrectly in the past? Is it to normalize data, so events like companies merging are able to run reports with pre-merge data?

This is a thorny subject, but it seems like one that we should be thinking about as data professionals, looking to give guidance to our clients.

Steve Jones

(Originally published at

The Voice of the DBA Podcasts