Everything Oracle

A Weekend Away

If you’ve browsed the articles on this site you’ll have observed that I often proclaim the business benefits of giving end users full control over their own data, the benefits that follow from ad hoc or “do it yourself” reporting. There is, of course, no shortage of unproven ideas for increasing business efficiency, and you may well be thinking that my advocacy of “putting the end user in control” is just another one of them. Well, here’s a little case study. It’s small in size, but you might find it interesting.

I was once asked by the CEO of a large corporate – who was far from convinced that the IT department’s plans to spend a large sum on a “new fangled” reporting system was justified – if I could provide him with any tangible evidence that changing the corporate culture to a “do it yourself” model would yield measurable benefits. In response to this challenge I proposed a little experiment to put the theory to the test. And, since funding the experiment was far cheaper than funding the new IT system, the CEO agreed.

We arranged for sixteen of the company’s senior executives, all of whom were key decision makers, to spend a “weekend away”, on one of those courses beloved of the world’s HR departments. Our executives arrived on a Friday evening, were ensconced in country house hotel – one providing facilities commensurate with their corporate standing – and after an evening of wining and dining all were sufficiently sober the following morning to act as “lab rats” for the experiment.

I first gave the candidates an introductory talk about the business that was to be the subject of experiment, a business that they were all unfamiliar with. I explained the terminology, and the factors that were important to success. Then I divided the candidates into two groups for a battery of tests and challenges to determine how well they had assimilated the information, and how they would perform in their new decision making roles with realistic sets of business data. One group, “the paper-bound” group, was given a set of standardized paper reports, the type that have been the backbone of business since the days scriveners first put quill pen to paper. The other group, the “free range” group, were escorted to the hotel’s IT suite where, with the help of a link to corporate headquarters, they were able to access an ad hoc query tool, one that they were already familiar with. The query tool gave them access to the same data set that had been used to derive the reports for their “paper-bound” colleagues. On the second day of the experiment a new business was selected and, in a cross-over design, the membership of the two groups was switched so that any inherent strength in one group would not bias the results of the experiment.

The executives were asked to perform various tasks that were representative of managing a business, reacting to new information, and making decisions. For example, using the historical data available to them, the executives were asked to determine as many salient issues about the new business as possible, such as trends in sales and threats from competitors. There were asked to list all the issues they discovered. And, most importantly, they were asked to record what they were doing with the data, such as what report or graph they were looking at, when they discovered each issue.

Now, I had carefully prepared the data sets so that they contained dozens of salient features, representing trends in sales figures, anomalies that might represent opportunities or threats from competitors. Some features were blindingly obvious, whereas others were subtle and could only be seen by viewing the data in a particular way. When the experiment was over the relative success of the “free range” group with respect to the “paper-bound” group was determined, and the factors that led to that success were analysed.

Side by Side

Conventional reports are restricted by their geometry: a certain column of values always occurs next to another column of values, and is always a number of columns away from some other column of values. Now I had deliberately placed some interesting columns which showed significant trends side and side, and others far apart. It was clear from the analysis that both groups made an effort to compare these columns as they were clearly relevant to the performance of the model business. Where the columns were side by side on paper both groups scored similarly, but where they were far apart on paper, the “free range” group performed much better. And the reason why they did so much better was clear from their notes: the “free range” group had constructed a customized report showing the two columns side by side.

This ability to control the manner in which data sets are compared with one another is one of the main benefits of ad hoc reporting. Typically, a measure, such as the total sales value, will depend on many different variables or dimensions. Selecting these dimensions two at a time makes this comparison much easier.

There are, of course, other ways to make comparisons that are often far better, but which OBIEE does not support at present. For example, a 3D graph, in which a measure is displayed as a surface showing its simultaneous dependency on two different dimensions is often very informative. And then there are more sophisticated techniques, such as multi-linear regression and principal component analysis, which can be used to determine the dependencies of a measure against dozens of dimensions simultaneously, and to display the relative strengths of these dependencies. But, sadly, at present OBIEE provides no support for this type of analysis either.

Serendipity

Anyone who has used an ad hoc query tool to analyse business data will be all too familiar with those little serendipities that lead to a better understanding of the business. An analysis of the executives’ notes revealed many instances where, when trying to construct a particular report to answer one question, an executive suddenly noted something odd or unusual in the data, something unrelated to the question that he or she was trying to answer, “It’s interesting that that value is so high at the end of the fiscal year”, or “I wonder if the graph for the industry average has a similar kink in the third quarter”.

These little discoveries by happenstance are a constant source of small hypotheses about the behaviour of a business. Subsequent analysis may, or may not confirm, that they are true. With a paper-based set of reports serendipities of this kind are rare, and without an ability to produce customized reports it is often impossible to follow them up directly – requesting the IT department the produce yet another hand-crafted report would take some time, and in cost terms would be difficult to justify, so the insight that the executive has just had is never followed up.

False Positives

There was one area, however, where the “free range” group did a little worse than their “paper-bound” colleagues.

If you go to the doctor feeling unwell then it’s quite possible that you’ll be told you have nothing to worry about, even though you are suffering from some life threatening condition. In the same way the “paper-bound” executives were less likely than their “free-range” colleagues to discover conditions that were life threatening to the model business that they were examining. However, when you go to the doctor there is also another possible outcome: the doctor may believe that you have a life threatening condition, and order some expensive treatment with serious side effects, when, in fact, no such condition is present. In business, these “false positives” are just as dangerous in terms of their impact on expenditure and on the health of the business.

In preparing the data sets I used a random number generator to produce the data which was subsequently superimposed on top of data which sometimes showed a trend and which sometimes did not. What was noticeable from the analysis was that it was quite common for the “free range” group to “discover” trends in the data where no such trends existed – the apparent trend was just due to random variation in the data. What was also interesting was that in almost all cases where these false positives were identified, the executives were examining the data using graphs (the “paper-bound” group had access to a small number of graphs, but the “free range” group exhibited a strong predilection for viewing the data in graphical form).

In particular, a little bit of statistical noise in the data at the beginning or end of the visible range on a graph was likely to be interpreted as a trend. And if this trend coincided with some other relevant business factor, such as a marketing campaign or the acquisition of a competitor, then the executives would conclude that this “ghost trend” was due to a causal link. Clearly, false positives can be very expensive – if, for example, on the basis of a trial marketing campaign that seems to improve sales, a national or regional campaign is launched at vast expense to the business.

This “tendency to see patterns where none exist” is a general characteristic of the way in which the human mind works. Biologists attribute this tendency to the costs and benefits to an organism of recognising, or failing to recognise, the presence of prey or predators – if that pattern in the undergrowth seems a little odd then you’re more likely to survive if you suspect that it may represent your dinner, and, even more so, if you suspect that you may represent its dinner! In any case, however it may have arisen, this bias is alive and well when it comes to analysing business data for potential sources of profit or loss.

The statistical techniques needed to avoid this bias and to quantify whether or not noisy data does, or does not, reveal a trend have been available for over a century, and there are plenty of statistical software packages available for the professional statistician. And in certain business applications, such as when analysing the effectiveness of a new pharmaceutical product, the costs and benefits to the business are such that an in-house statistician will always be available to carry out the analysis.

Now, what is required in the next phase of business re-engineering is to make routine use of some of these statistical techniques in the everyday analysis of business data. This will not be an easy task. Even in the sciences where individuals are generally more numerate than the average business executive, the mis-application of statistical techniques is woefully common. That said, the introduction of some simplified statistical techniques would do less damage to business decision making than simply following the status quo.

Unfortunately, OBIEE offers no help in this regard at present. For example, OBIEE should provide a facility to fit a smooth curve between the points on a graph and then plot error bars or a confidence region on either side of the curve to give the user a feeling for the level of uncertainty in the data. It should provide a facility to say whether or not there is a trend in the data being graphed by automatically calculating a level of significance. A minimalist adventure by OBIEE into the world of statistics would give executives some feeling for the uncertainty in the data they are analysing with only a modicum of additional training.

The Results

So what was the overall outcome of this experiment? Well, when averaged across all measures of performance, the “free range” group did close to 30% better than their “paper-bound” colleagues. Think about the implications that this might have in giving your business that competitive edge!

Everything Oracle

Home

Everything Oracle

Ad	Hoc	Reporting

The Business Benefits