de l’information
The Confounding Effect of Class Size on
The Validity of Object-Oriented Metrics
Khaled El Emam
National Research Council, Canada
Institute for Information Technology
Building M-50, Montreal Road
Ottawa, Ontario
Canada K1A OR6
khaled.el-emam@iit.nrc.caSaida BenlarbiNishith GoelCistel Technology210 Colonnade RoadSuite 204Nepean, OntarioCanada K2E 7L5{benlarbi, ngoel}@
Abstract
Much effort has been devoted to the development and empirical validation of object-oriented metrics.
The empirical validations performed thus far would suggest that a core set of validated metrics is close
to being identified. However, none of these studies control for the potentially confounding effect of class
size. In this paper we demonstrate a strong size confounding effect, and question the results of previous
object-oriented metrics validation studies. We first investigated whether there is a confounding effect of
class size in validation studies of object-oriented metrics and show that based on previous work there is
reason to believe that such an effect exists. We then describe a detailed empirical methodology for
identifying those effects. Finally, we perform a study on a large C++ telecommunications framework to
examine if size is really a confounder. This study considered the Chidamber and Kemerer metrics, and
a subset of the Lorenz and Kidd metrics. The dependent variable was the incidence of a fault
attributable to a field failure (fault-proneness of a class). Our findings indicate that before controlling for
size, the results are very similar to previous studies: the metrics that are expected to be validated are
indeed associated with fault-proneness. After controlling for size none of the metrics we studied were
associated with fault-proneness anymore. This demonstrates a strong size confounding effect, and
casts doubt on the results of previous object-oriented metrics validation studies. It is recommended that
previous validation studies be re-examined to determine whether their conclusions would still hold after
controlling for size, and that future validation studies should always control for size.
1 Introduction
The validation of software product metrics has received much research attention by the software
engineering community. There are two types of validation that are recognized [48]: internal and external.
Internal validation is a theoretical exercise that ensures that the metric is a proper numerical
characterization of the property it claims to measure. External validation involves empirically
demonstrating that the product metric is associated with some important external metric (such as
measures of maintainability or reliability). These are also commonly referred to as theoretical and
empirical validation respectively [73], and procedures for achieving both are described in [15]. Our focus2in this paper is empirical validation.
Product metrics are of little value by themselves unless there is empirical evidence that they are
associated with important external attributes [65]. The demonstration of such a relationship can serve
two important purposes: early prediction/identification of high risk software components, and the
construction of preventative design and programming guidelines.1
Some authors distinguish between the terms ‘metric’ and ‘measure’ [2]. We use the term “metric” here to be consistent with
prevailing international standards. Specifically, ISO/IEC 9126:1991 [64] defines a “software quality metric” as a “quantitative scale
and method which can be used to determine the value a feature takes for a specific software product”.
21 Theoretical validations of many of the metrics that we consider in this paper can be found in [20][21][30].