Mistakes can come in many flavors, errors of commission and omission; calculation mistakes or errors in mathematics (wrong formulas, logic or just ignoring things like co-variance), and just plain stupid mistakes. The group as a whole is the single biggest reason Good Numbers Go Bad. Mistakes by definition occur by accident and are not driven by direct animus. The grace and speed in which you recognize and recover from a mistake will determine the long-term prognosis of the practitioner and his/her program (assuming you don’t make the same mistake more than once or twice). Ignoring a mistake is bad practice; if you need to make a habit of brazening out the impact of mistakes, you should consider a new career as you have lost the long term battle over the message.
Collection mistakes are a category that covers a lot of ground ranging from gathering wrong data to erratic data collection. While collecting the wrong information can lead to many other kinds of mistakes, we will explore credibility issues in this section. Recognition and the recovery from collection errors which lead to credibility issues will be explored in depth in this section.
“In order to capture metrics, the procedures, guidelines, templates, and databases need to be in sync with the standard practices.”
— Donna Hook, Medco
Data collection errors typically represent errors of omission (data not collected); however, occasionally the wrong information is collected or data is not collected at all. Collecting the wrong data (or data you do not understand) will create situations where your analysis will be wrong (garbage in) with the possibly that you won’t know it (gospel out). Someone will usually discover this error at the worst possible time, leading to profuse sweating and embarrassment. Gathering the wrong or incomplete data is a non-trivial mistake which makes Good Numbers Go Bad. However, what you do about it will say a lot about your program. Begin by making sure you have specified the data to a level that allows you to ascertain that what you collect is correct. Audit the collection process against the collection criteria periodically helps to make you collect the correct data and collect it correctly. Create rules (or at least rules of thumb) that support validation. Rules of thumb will help you to quickly interpret the data. Did you get the quantity of data you expected? Has the process capability apparently changed more than you would reasonably expect?
Measures and metrics can be perceived to be so important that panicked phone calls are known to precede collection. Equally as interesting are the long periods of silence that occur before the panic. Erratic data collection sends a message that the data (and therefore the results) are only as important as whoever goosed the caller (or slightly less important to whatever the caller was doing right before he/she called). Inconsistent collection leads to numerous problems including rushed collection (after the call), mistakes and an overall loss of face for the program (fire drills and metrics ought to be kept separate). Consistency spreads a better message of quiet importance that can supplant the urgency of yelling.
“We accidentally used one number instead of a correct value. Now our stakeholders ask for a second source.”
— Rob Hoerr, Fidelity Information Services
“Mathematical mistakes happen! We are all human!” The excuses are anthem, which means all measurement programs must take the time and effort to validate the equations they use. Equations must be mathematically and intellectually sound. Inaction in the face of mistakes in the equations or results makes Good Numbers Go Bad. If a mistake is found, neither results nor equations should be ingrained to the point of freezing your project into inaction. This places a lot of stress on the need to create measurement and metrics specifications. Once the specification, (specifications include data like a description, formulas and definitions) is created, it is easier to make sure you a measuring what you want and that you get the behavior you anticipate. The spec provides a tool to gauge the validity of the math, the validity of the presentation, and, by inference, the validity of the analysis.
Statistics has long been a staple of graduate business schools, which instill the belief that numbers can prove anything. Numbers, however, require a sensitivity to the equations that flies in the face of this mentality. When simple relationships are ignored to make a point Good Numbers Go Bad. Examples of questionable math can include graphs with the same variable (in different forms) on both axes presented with linear regressions lines driven through them. The created co-variance goes unrecognized, leaving the analysts speculating on what the line means without the recognition that the relationship is self-inflicted. Developing a simple understanding of the concepts of co-variance, ‘r”-squared values and standard error are easy steps to help sort out basic conceptual errors. A corollary to this is that the knowledge of statistics will not necessarily stop the mistake of adding the wrong EXCEL cells together, but it can’t hurt. Always check your equations, your statistics, and never fail to check the math!