One of the most useful things I’ve learned from working with data is to look for what’s missing.
I don’t mean missing data in the traditional sense. I mean stepping back from what’s in front of me and asking whether we’re even looking at the right things.
When I’m building or reviewing a report or dashboard, some of the most important questions I ask are:
- How is this supposed to be actionable?
- What decision am I trying to influence, inform, or support?
- Who is the intended audience or information consumer?
- What would someone do differently if this number changed?
And then: What’s missing?
That last question has saved me a lot of time over the years. More importantly, it has occasionally helped me realize we were solving the wrong problem.
Eight Million Records to Get 8,000
Years ago, I was contracted to rewrite a collection of Crystal Reports into Microsoft Reporting Services. The decision had already been made before I arrived. The client believed the existing queries were inefficient and wanted them rewritten as stored procedures. The assumption was that stored procedures and the newer reporting platform would perform better.
They were both right and wrong.
There was definitely a performance problem. Some queries took three or four hours to run, consumed an enormous amount of server resources, and occasionally failed before the report could even display. But the important problem wasn’t Crystal Reports versus Reporting Services. It wasn’t even SQL queries versus stored procedures. It was where the filtering occurred.
The queries were returning roughly 8 million records. The report then filtered those records down to what the user actually needed: about 8,000 records. We were moving and processing millions of records just so the reporting software could throw almost all of them away.
Once I moved the filtering into the query, processing time dropped from hours to about ten minutes. The reports stopped draining the server and displayed reliably.
The technology mattered. But it wasn’t the important question. The important question was the one nobody was asking.
Three Hours Became Fifteen Minutes
Another client had a report requiring someone to spend about three hours reviewing it every morning. The purpose was to identify displays that needed attention so representatives could be sent out to fix them.
My question was pretty simple: What makes something an exception? What is too high? What is too low? What actually requires someone to do something?
Once the client could define those conditions, there was no reason for a person to spend three hours looking through everything that was normal. I created an exception report, and the morning process went from about three hours to about fifteen minutes.
Today, exception reporting isn’t particularly novel. At the time, around 2010, very few people I encountered were doing it. The important part wasn’t the technology anyway. It was changing the question.
Instead of asking, “How can we make it easier to review all this data?” we asked, “Why are we making someone look at data that requires no action?”
That question still influences how I think about dashboards today. If a number changes, what would you do differently? If the answer is nothing, I start wondering why we’re asking someone to look at it.
The Report Nobody Missed
Sometimes what’s missing isn’t a number at all.
At one company, users were supposed to pull a report every workday, look through six tabs, and find anomalies. I argued with my boss that nobody was doing it; they were too busy to pull a report every morning and do that much searching for something that was usually within normal parameters. He insisted they were doing it.
Then the report stopped working. For two weeks. Nobody contacted me.
That seemed like a pretty good piece of missing information.
I redesigned the report. Instead of requiring users to go find it, a one-page PDF containing the important information was emailed directly to the distribution list. Push instead of pull. One page instead of six tabs.
About a month later, the report failed again. I had six messages asking about it before I even got to my desk.
The original report wasn’t necessarily missing information. It was missing its audience. We sometimes spend enormous amounts of effort deciding what information belongs in a report and surprisingly little time thinking about the person who is supposed to use it.
A report nobody reads can be perfectly accurate and completely useless.
Sometimes the Missing Information Is on the Other Side
Eventually I realized that looking for what’s missing isn’t only useful when I’m creating reports. It’s also useful when someone else is presenting information to me.
Around 2005, a United Way representative came to the company where I worked as part of an annual charity drive. During the presentation, she said United Way had helped 2 million people the previous year.
At the time, Minnesota had a population of roughly 5 million.
Something didn’t add up. If she meant United Way had helped 2 million people in Minnesota, that would have been around 40 percent of the entire state population.
I didn’t know what was wrong with the number. Maybe nothing was. Perhaps 2 million referred to people nationally rather than in Minnesota. Maybe it represented services provided rather than unique individuals. Maybe one person receiving help from two programs was counted twice. Maybe “helped” had a definition I didn’t understand.
The problem wasn’t necessarily the number. The problem was the missing context.
And I think that distinction matters. Looking for what’s missing doesn’t mean assuming people are lying to you. It doesn’t mean distrusting every statistic or looking for some sinister motive behind every chart.
It means asking whether you have enough information to reach the conclusion you’re being invited to reach.
That’s skepticism. It doesn’t have to become cynicism.
The Question Behind the Question
I don’t remember when I started saying “look for what’s missing.” It wasn’t a technique someone taught me. It’s simply a name I eventually gave to something I learned through a lot of experiences, both good and bad.
Sometimes what’s missing is another piece of data. Sometimes it’s context. Sometimes it’s the intended audience or the action someone is supposed to take. Sometimes we’re measuring the wrong thing.
And sometimes the biggest thing missing is the question nobody thought to ask.
We have access to more data than ever before. That doesn’t necessarily mean we understand more.
The next time a report, dashboard, presentation, news story, salesperson, chart, or even your own analysis seems to point toward an obvious conclusion, don’t automatically assume it’s wrong.
Just remember to ask:
What’s missing?
Erik Brown 