Assessing the benefits of technology

No one would question that in the last 50 years, technology has had a significant impact on reducing costs and increasing worker productivity. At each step, from the earli-est mainframes to the cloud, increased productivity and re- duced costs have been the clear drivers of technology adoption.

What hasn’t been so clear is how to credibly quantify these benefits when building an ROI business case for a technolo-gy decision. I’ve seen far too many decisions based on little more than gut feel, the desire to reduce risk or the end of support for a version of software. With a proper business case, results can be compared between projects and organizations can make rational decisions. As the econ- omy tightens and competition increases, making good technol- ogy decisions and being quick to discard bad ones is a competitive advantage organizations can’t overlook. But this relies on a fair measurement of the benefits.
Whether you’re a vendor describing the benefits of your solution or an organization estimating the benefits you
will achieve from a technology purchase, understanding the dif- ference between a strong benefit and a weak benefit is critical. At my company, we find that most people, when asked, will broadly categorize benefits as hard or soft. That might work in some cases, but there may be a better framework for assessing the value of a benefit.
Rethinking How We Con- sider Benefits A business case is an estimate built on a series of smaller esti- mates that generate that result. Each of these mini estimates hinges on two factors: believ- ability and, to a far lesser extent, variability. In most cases, these two factors are correlated, with an estimate having a wide vari- ation being less believable. It’s not hard to see that the believ- ability of the business case is built on the cumulative believ- ability of each benefit — a nice way of saying the more fiction in the benefits, the less believable the final business case.
In reality, there are four types of benefits, not two. Let’s create a new framework that ranks benefits based on believability. We’ll use four categories to give us an idea of a strong benefit versus a dubious one.

First-Order Benefits
A first-order benefit is a benefit that will absolutely be achieved — without doubt about the outcome or the amount. If moving to the cloud will elimi- nate the maintenance cost on a server, the amount of which is budgeted and known, there’s a 100% chance that benefit will be
achieved. Clearly, this is a strong benefit and has 100% believabil- ity in the business case.

Second-Order Benefits
A second-order benefit is es- sentially a first-order benefit that includes a hedging word. You can recognize it when a benefit includes words such as “expect” (or “plan,” “intend,” “hope,” etc.). For example, a benefit might be described as: “By mov- ing half of our applications to the cloud, we should be able to reduce the energy costs by 50%.” That benefit is very likely true, but the amount is also likely to vary a bit. The benefit is believ- able and is strong, but it is not
as strong as a first-order benefit. We consider that type of benefit as 70% believable.

Third-Order Benefits
A third-order benefit is an in- crease in worker productivity. It starts with two quick tests before you estimate the value of the ac- tual benefit. First, do you believe in a broad sense that there is a benefit? And second, does the technology directly touch the user achieving that benefit? Let’s say you are trying to estimate the value of a new smartphone app for your sales team. The first test would be to look at the app and see if you agree that it delivers value. Likely true. The second test is that the app directly impacts the user. This is true because the salesperson holds the smartphone in their hand and directly interacts with the app. That makes it a third-order benefit. How do we calculate the value of a third-order benefit? It’s not as hard as it seems. Start by recognizing that all pro- ductivity benefits follow a bell curve. Users will achieve more or less, but there’s no perfect number. Surveying users after showing them the application,
measuring the time taken during a workflow and asking a ven- dor’s reference customers about their experiences are all reason- able approaches to making an estimate. We’ve found there are often at least two ways to gener- ate an estimate for a productivity gain. Still, given the high degree of variability, we tend to put a 40% believability number on any third-order benefit.

Fourth-Order Benefits
A fourth-order benefit is a dis- connected benefit, one with mul- tiple steps from the technology to the benefit, and it includes joining words such as “and”
or “then.” If a benefit starts to sound like a rambling story, it’s a fourth-order benefit. For ex- ample, one cloud provider insist- ed its application would increase profit margin by 2% and gener- ate 10% more new products each year based on nothing more than one customer’s claims. Another vendor insisted it would increase average basket size for e-com- merce sites by 20% based on a loose link between website mon- itoring (what it did) and a report on the internet that tied website performance to buying behav- ior. A fourth-order benefit has effectively no believability and is best ignored when developing a business case.
The strength of a business case is built on the strength
of the benefits that drive its results. Generating a high ROI on weak benefits is more likely to damage your case rather than strengthen it. Using the four orders as a framework, you can quickly understand the strength of your proposal and assess your likelihood of success. In most cases, it’s better to discard weak benefits that do more to under- mine believability than add to your ROI.