Question

1. One recurring idea in software engineering research is using various kinds of data to try to identify "problematic" areas of a codebase: classes/methods that seem to have notably higher defect density (more bugs/lines of code) than elsewhere, areas that tend to be tied to flaky tests (those that fail nondeterministically for no immediately apparent reason), etc. How might you try to gather such data for a project using version control and issue trackers? What are some of the major caveats of the approach you're suggesting? (Using this kind of data is inherently heurstic, but can still provide useful results, with caveats.)
2. Consider some problematic ways to use this data.
   • One classic example is to add up the lines of code written by a developer in some period of time, based on changes in version control, and use that to measure "productivity" (and thus, to influence raises and promotion). There are all sorts of other problems like this:
   • More code is not necessarily better (maybe it's longer because it's poorly written)
   • Does removing code count negatively? (If so, anyone unlucky enough to be the one who deletes dead code from a removed feature is in for a bad performance review.)

What are other problematic ways this data could be misused to draw incorrect conclusions, and why are they problematic?

Answer 1

Misleading statistics are simply the misusage – purposeful or not – of a numerical data. The results provide a misleading information to the receiver, who then believes something wrong if he or she does not notice the error or the does not have the full data picture.

Given the importance of data in today’s rapidly evolving digital world, it is important to be familiar with the basics of misleading statistics and oversight. As an exercise in due diligence, we will review some of the most common forms of misuse of statistics, and various alarming (and sadly, common) misleading statistics examples from public life.

Here are a few potential mishaps that commonly lead to misuse:

• Faulty polling

The manner in which questions are phrased can have a huge impact on the way an audience answers them. Specific wording patterns have a persuasive effect and induce respondents to answer in a predictable manner. For example, on a poll seeking tax opinions, let’s look at the two potential questions:

– Do you believe that you should be taxed so other citizens don’t have to work?
– Do you think that the government should help those people who cannot find work?

• Flawed correlations

The problem with correlations is this: if you measure enough variables, eventually it will appear that some of them correlate. As one out of twenty will inevitably be deemed significant without any direct correlation, studies can be manipulated (with enough data) to prove a correlation that does not exist or that is not significant enough to prove causation.

• Data fishing

This misleading data example is also referred to as “data dredging” (and related to flawed correlations). It is a data mining technique where extremely large volumes of data are analyzed for the purposes of discovering relationships between data points. Seeking a relationship between data isn’t a data misuse per se, however, doing so without a hypothesis is.

• Misleading data visualization

Insightful graphs and charts include very basic, but essential, grouping of elements. Whatever the types of data visualization you choose to use, it must convey:

– The scales used
– The starting value (zero or otherwise)
– The method of calculation (e.g., dataset and time period)

• Purposeful and selective bias

The last of our most common examples for misuse of statistics and misleading data is, perhaps, the most serious. Purposeful bias is the deliberate attempt to influence data findings without even feigning professional accountability. Bias is most likely to take the form of data omissions or adjustments.

• Using percentage change in combination with a small sample size

Another way of creating misleading statistics, also linked with the choice of sample discussed above, is the size of said sample. When an experiment or a survey is led on a totally not significant sample size, not only will the results be unusable, but the way of presenting them – namely as percentages – will be totally misleading.