In Search of Statistical Understanding

Mark Twain once said, “…There are three kinds of lies: lies, damned lies and statistics.” Unfortunately most people seem to have taken that statement to heart, shunning the usefulness of statistics in risk management and decision making by either not using them or not even bothering to learn proper statistics. The dearth in use of statistical information and analysis by the general public has resulted in the common misrepresentation of various pieces of information due to a lack of sufficient reported parameters. This misrepresentation has created scenarios where inappropriate decisions were favored over more rational decisions creating instances of inefficiency in already difficult situations. These scenarios are most commonly demonstrated in, but not limited to, the multitude of opinion polls that are conducted on a daily basis which supposedly dictate public policy.

Unfortunately statistical misrepresentation also infiltrates other severe issues such as medical decision-making. These misrepresentations stem either from a lack of understanding regarding how statistical theory actually operates or a deliberate attempt to boost or lower the success rate of a particular product/treatment and are perpetrated by patient, physician and pharmaceutical company. The chief concern among the public should be that inaccurate statistical analysis of these procedures at best results in a significant waste of time and money and at worst results in the greater probability of a loss of life/lives. Such a lack of statistical application is made worse by the fact that all of the relevant information is easily available, but simply not interpreted properly. Without using an objective statistical analysis how is one able to discern the difference between one procedure/product vs. another? Testimonials are rarely an appropriate determining agent due to the real possibility for a conflict of interest. Overall it is troubling that there is such a lack of importance placed on an issue that would eliminate waste at almost no additional cost and carries a high probability of saving lives.

Cancer screening and their associated false positives are a very common example of where ‘common sense’ and genuine statistical analysis part ways when coming to a conclusion regarding the result. For example the generic example used many times to illustrate this point is: if there is only a 1% chance of a women having breast cancer and a mammogram has a 90% rate of accuracy at detecting cancer in an individual that has cancer and a 9% chance of recording a false positive (the mammogram detects cancer in an individual without cancer) there is only a 9.9% chance that a positive mammogram will actually identify an individual with cancer. Such a low result is shocking to one’s natural intuition when considering only a 9% rate of false positive vs. a 90% rate of accuracy at detecting cancer, so why is 9.9% the correct result?

The basic explanation for the ‘shock’ comes from minimizing the importance of the original probability that a woman has cancer. The result is easy to understand when comparing the false positive probability rate to the actual occurrence rate. The false positive rate is nine times larger than the actual cancer occurrence rate, thus for a 100% accurate test with regards to detecting cancer in an individual with cancer there would be only a 10% chance that a positive test resulted in actually detecting cancer in an individual. In the above example the test accuracy was 90% thus there is only a 9.9% chance. Basically even with zero statistical understanding thinking about the issue properly leads one to conclusion that the correct answer needs to be somewhere in the neighborhood of the test accuracy being 10% due to the ratio between the false positive and the real positive. So in the end ‘common sense’ actually does coincide with statistical analysis as long as the ‘common sense’ used is legitimate. The sad thing is that even many physicians are surprised by this result despite the fact that they should be more in tune to such statistics.

So if there are many advantages to using statistics when making a decision, why do so many individuals elect not to use statistics? The most obvious answer is the inherent bias most individuals have towards mathematics and math related subject matter. Statistics inhabit the world of math and as a whole the part of the world they exist in is not the happy easy arithmetic neighborhood, but the difficult formula and theory neighborhood. Therefore, the application of statistics takes significant and real effort over simply punching a few numbers into a calculator; this required applied effort is another strike against statistics in a world where all things are desired to be fast and simple. The fact is that statistical analysis actually makes difficult decisions easier if used appropriately.

Another obstacle that may reduce the probability for the application of statistics in a decision-making process is a lack of certainty. Statistics do not generate a prediction of what will happen, but of what will most likely happen. Unfortunately this reality of statistics conflicts with the general psychological map that most people possess. Most individuals do not think in the context of an event happening 100 times and the probability associated with what happens each time over those 100 samples. Instead individuals focuses only on the single time that he/she will experience the particular event. This expectation leads to more trust in instinct (gut feeling) than statistics. This mindset is unfortunate because statistics exist due to the omnipresence of variability in existence including events beyond instinct.

A secondary aspect to this separation between statistics and certainty is a misunderstanding of statistics in general. Statistics generate a probability of occurrence for different possibilities over many different repetitions of the same general event. However, because a lot of event in general life do not have significant periods of repetition individuals tend to view the outcomes of those events as the actual probability of occurrence rather than what statistics predict. Basically the fact that a particular outcome only has a 3% probability of occurrence in a given scenario over 1000 tests will have little influence in the mindset of an individual that experiences that outcome 2 of the 3 times that scenarios has occurred in real life. Thus such an experience may lead an individual to doubt the accuracy and/or importance of statistics in other aspects of existence, thus leading to the incorrect viewpoint that statistics are a waste of time and effort.

In fact the power of the statistical method may also turnoff individuals because even when they choose to use statistics they can easily be disappointed by the overall power of the test because their intuition tells them the result should be more meaningful. For example suppose a brokerage firm wants to determine which of their 25 employees have been performing the most efficiently. An evaluation test is created that can identify the best performing employee with 97% accuracy. Based on statistical theory what is the actual probability that the best performing employee will be identified? Using Bayes’ theorem the evaluation test identifies the best performing employee 57.4% of the time. Although correct statistically, it does not sit well if the typical person that a test that is initially believed to have an accuracy rate of 97% in actuality only has an accuracy rate of 57.4%.

A third concern relates back to the aforementioned problem regarding available information and raises its own chicken vs. egg question. Certainly not all relevant information is going to be available to a decision-maker at the time of the decision. However, it does behoove an individual to have as much relevant information as possible regarding the issue. Unfortunately this belief does not appear to be the attitude of major polling groups and the news media as they present extremely simplified questions without any expansive circumstances. This behavior raises the question of do these polling groups behave like this because they believe that the public want simplicity and would not use additional information or do they behave like this because they are lazy and/or incompetent and cannot ask important qualifiers to their questions? This question is important because if the public learns to value statistics in decision-making then one can better assess the probability that polling groups will change their behavior when collecting and presenting information to include more details.

The reason supplemental information and qualifiers are important is because issues are rarely as simple as polling questions suggest they are. For example the most common poll question in the recent healthcare debate was ‘do you support a public option?’ Wow, what an amazingly simple question overlaying a complex issue. The first error in the question is that of the ignorant respondent. The pollster assumes that each individual answering the question has relatively the same definition for what entails a ‘public option’, which is highly unlikely. In similar fashion the pollster assumes that each individual is aware of the definition for the term being used by those in Congress. Also the pollster does not inquire to the details surrounding the success or failure of such an issue. Basically what the respondent would gain or loss if a public option existed or did not exist. None of the elements that go into creating a public option and how they would influence the answer of the respondent are discussed which defeats the point of even asking the question.

The importance of these qualifiers can be seen in the following example. Suppose you ask the following question to 1000 people: ‘Would you like 10 dollars?’ It would be very surprising if any one of the respondents answered in the negative. However, what if an important piece of information, which was excluded from the first go around, was added and another 1000 people were asked this question: ‘Would you like 10 dollars which I just stole from that 5-year old girl over there who is still being a baby and crying about it?’ Adding the information regarding the origin of the 10 dollars, another layer of complexity to the question, has changed the question dynamic completely. Now it would not be surprising if the level of response flipped to an overwhelming ‘no’. What if instead of a 5-year old girl the money was stolen from a billionaire, how would that shape the response curve?

Another problem with the media outlets and the way they diminish the importance of statistics is inappropriate presentation of growth or decline percentages. Typically this information is presented as relative changes without illustrating the absolute numbers that represent those changes (absolute changes). Not looking at the absolute changes can lead an individual to radically erroneous conclusions. For example suppose from year x to year y it is reported that the GDP in a given country increases by 25% under President A whereas 5 years ago the GDP increased by only 5% under President B. Clearly President A must be doing a better job working with Congress to manage the economy right? Not necessarily as the GDP 5 years ago could have been 3 trillion whereas in year x the GDP was 400 billion. When looking at the absolute numbers the increase in GDP 5 years ago was 150 billion whereas the increase in GDP from year x to year y is only 100 billion. So despite a 5x increase in percentage between the two equal distant time periods, the actual increase 5 years ago was 1.5x larger than the increase from year x to year y.

Reporting the absolute change is always better than the relative change because as described above, one can calculate the relative change from the absolute change, but cannot calculate the absolute change from the relative change. Unfortunately despite the above example relative changes are almost always going to be a larger number vs. absolute changes and the media in its ever expanding effort to attract more public attention over actually informing the public grab the relative change number to make the headline more important than it might actually be.

Clearly there are obstacles that need to be overcome before statistics can be implemented on a large scale. Fortunately most of these obstacles revolve around misinformation rather then difficulty of understanding. This characteristic is favorable because misinformation does not tie to intelligence, but communication and familiarity. Basically one does not need an advanced level of intelligence to understand and apply statistics.

At its heart statistics focuses on a search and discovery of patterns with later a deduction of any significant meaning to those patterns and how they may impact future events. The problem is that it tends to be difficult to perform such an exploration and analysis methodology without a proper level of experience. This lack of experience is telling in that most people are exposed to their first significant statistics course, if they are ever exposed to one in the first place, in college. It is true that the concept of probability is frequently introduced earlier than college, but in most instances such introduction does not discuss statistics and its importance in sufficient detail. College exposure is typically far too late if a goal is to develop an appreciation and understanding of statistics and what role it plays in real life. Heck, most college individuals that take statistic course lament the fact that they have to take it for their given major.

One reason for why exposure to statistics occurs at such an advanced age is that most believe that a strong core of mathematics is required before beginning study in statistics otherwise the effort applied to learn statistics will be wasted due to a lack of understanding in general mathematical theory. Unfortunately this thought process is not entirely accurate because although the study of statistics does involve advanced concepts in mathematics, there are other critical aspects to understanding the nature behind the results produced by statistical formulas.

For example one forgotten aspect of statistics is exploratory data analysis (EDA), which seeks to identify what the data is saying, not necessarily how it was calculated. EDA is an important aspect of understanding statistics because one needs to understand the context of the numbers that enter into and are spit out by statistical formulas. Also EDA focuses on using graphical information instead formula and theory which make it easier to younger students to both enjoy and understand. The application of EDA allows statistical analysts to understand why certain data should not be considered relevant for a particular statistical analysis for the inclusion of outliers or irrelevant/inappropriate data generates errors in the end result. EDA leads to the understanding of why a question like ‘what are the flaws in the methodology used for data collection’ is important to ask and how to properly answer it.

Such early analysis experience can be taught at an early level by giving students a list of data sets and details on how that data was generated and asking which sets are accurate, which sets are trash and which sets are usable as long as certain steps are taken to ensure accuracy. Also students can be asked to comment on the relevance of the outcome for certain statistical tests on given sets of data without having to do the tests themselves. Thus as a first step in renewing statistical thought in society, it would go a long way to improving the attitude individuals have towards statistics if statistical reasoning were taught before statistical theory and formulas, the mindset of statistics before the math. The issue of teaching statistics is especially pertinent to education reform. If the point of education is to ensure a populous that has the ability to reason and communicate effectively to each other in society then teaching and applying statistical reasoning is essential to achieving this goal.