Experimentation in Software Engineering: An Introduction / Edition 1

by Claes Wohlin, Per Runeson, Martin Host, Magnus C. Ohlsson, Bjorn Regnell

Overview

The purpose of Experimentation in Software Engineering: An Introduction is to introduce students, teachers, researchers, and practitioners to experimentation and experimental evaluation with a focus on software engineering. The objective is, in particular, to provide guidelines for performing experiments evaluating methods, techniques and tools in software engineering. The introduction is provided through a process perspective. The focus is on the steps that we go through to perform experiments and quasi-experiments. The process also includes other types of empirical studies. The motivation for the book emerged from the need for support we experienced when turning our software engineering research more experimental. Several books are available which either treat the subject in very general terms or focus on some specific part of experimentation; most focus on the statistical methods in experimentation. These are important, but there were few books elaborating on experimentation from a process perspective, none addressing experimentation in software engineering in particular. The scope of Experimentation in Software Engineering: An Introduction is primarily experiments in software engineering as a means for evaluating methods, techniques and tools. The book provides some information regarding empirical studies in general, including both case studies and surveys. The intention is to provide a brief understanding of these strategies and in particular to relate them to experimentation. Experimentation in Software Engineering:
An Introduction is suitable for use as a textbook or a secondary text for graduate courses, and for researchers and practitioners interested in an empirical approach to software engineering.

Product Details

ISBN-13:	9780792386827
Publisher:	Springer-Verlag New York, LLC
Publication date:	12/31/1999
Series:	International Series in Software Engineering Series , #6
Edition description:	Older Edition
Pages:	228
Product dimensions:	6.20(w) x 9.30(h) x 0.80(d)

About the Author

CLAES WOHLIN is a professor of software engineering at Blekinge Institute of Technology. His research interests include empirical methods in software engineering, software processes and software quality. He is a member of the Royal Swedish Academy of Engineering Sciences and a senior member of the IEEE. PER RUNESON is a professor of software engineering at Lund University. His research interests include empirical research on software testing and quality, and methods for such research. Prior to this, he worked as a consulting expert in industry, and he is a senior member of the IEEE. MARTIN HOST is a professor of software engineering at Lund University. His research interests include software quality and software process improvement, as well as empirical research methods in software engineering. MAGNUS C. OHLSSON is a quality assurance specialist at System Verification AB in Malmo, focusing on process improvement. His primary field of interest is verification and validation, with an emphasis on achieving proper quality efficiently throughout every step of the development process. BJORN REGNELL is a professor of software engineering at Lund University. His research interests include software requirement engineering, software product management, and empirical research on software engineering. ANDERS WESSLEN is a senior system architect at ST-Ericsson in Lund. His focus is on system-wide architectures, as well as system design and requirements.

Read an Excerpt

1 Introduction

The information technology revolution has meant, among other things, that software has become a part of more and more products. Software is found in products ranging from toasters to space shuttles. This means that a vast amount of software has been and is being developed. Software development is by no means easy; it is a highly creative process. The rapid growth of the area has also meant that numerous software projects have run into problems in terms of missing functionality, cost overruns, missed deadlines and poor quality. These problems or challenges were identified already in the 1960's, and in 1968 the term "software engineering" was coined with the intention of creating an engineering discipline that focused on the development of software systems.

Software engineering is formally defined by IEEE [IEEE90] as "software engineering means application of a systematic, disciplined, quantifiable approach to development, operation and maintenance of software". Software engineering in general is presented and discussed in books as, for example, [Sommerville95, Pfleeger98]. The objective here is to present how empirical studies and experiments in particular fit into a software engineering context. Three aspects in the definition above are of particular importance here. First, it implies a software process through pointing at different life cycle phases; secondly, it stresses the need for a systematic and disciplined approach; finally, it highlights the importance of quantification. The use of empirical studies is related to all three of them. The software engineering context is further discussed in Section 1.1. The need to turn software engineering more scientific and how empirical studies play an important role in this is discussed in Section 1.2.

1.1 Software engineering context

A software process model is used to describe the steps to take and the activities to perform when developing software. Examples of software process models are the waterfall model, incremental development, evolutionary development and the spiral model. These and other models are discussed in the general software engineering literature. A simplistic view of the software process is shown in Figure 2. It should be noted that the process is crucial whether we work with development of a new product or maintenance of an existing product.

In Figure 2, an idea and resources, primarily in the form of people, are inputs to the software process, and the people develop a software product going through the different steps and performing the different activities in the software process.

The development of software products is many times a complex task. Software projects may run over a long period of time and involve many people, due to the complexity of the software products that are developed. This implies that the software process often also becomes very complex. It consists of many different activities and many documents are written before the final product can be delivered. The complexity of the software process means that it is difficult to optimize it or even find a good enough process. Thus, it is important for companies to strive to improve their way of making business if they intend to stay competitive. This means that most companies are continuously trying to improve their software process in order to improve the products, lower the cost and so forth. The software process stresses the need for a systematic and disciplined approach of working. This approach is also needed when improving the software process, and hence a process improvement process is needed.

An example of an improvement process tailored for software development is the Quality Improvement Paradigm (QIP) [Basili85]. It consists of several steps to support a systematic and disciplined approach to improvement. The QIP is presented briefly in Section 2.6.2. A more general improvement process is the well-known Plan/Do/Check/Act cycle [Demming86]. The improvement processes include two activities, although the same terminology is not always used, that we would like to highlight:

• Assessment of the software process.

• Evaluation of a software process improvement proposal.

The assessment is conducted to identify suitable areas for improvement. Several models exist for assessing the software process. The most well known is probably the Capability Maturity Model (CMM) from Software Engineering Institute at Carnegie-Mellon University, USA [Paulk93]. The assessment models help in pinpointing where improvements are needed. The CMM has five maturity levels with so called key process areas on each level. It is recommended that companies focus on improvement areas according to their maturity level.

Assuming that it is possible to identify areas for improvement through some form of assessment, the next step is to determine how these areas of improvement may be addressed to cope with the identified problems. For example, if too many defects are found in system testing, it may be possible to improve earlier testing, inspections or even specific parts in the development, for example, software design. The objective is that the assessment of the current situation and knowledge about the state-of-the-art should result in that concrete process improvement proposals can be identified. When the improvement proposals have been identified, it is necessary to determine which to introduce, if any. It is often not possible just to change the existing software process without having more information about the actual effect of the improvement proposal. In other words, it is necessary to evaluate the proposals before making any major changes.

One problem that arises is that a process improvement proposal is very hard to evaluate without human involvement directly. For a product, it is possible to first build a prototype to evaluate if it is something to work further with. For a process, it is not possible to build a prototype. It is possible to make simulations and compare different processes, but it should be remembered that this is still an evaluation that is based on a model. The only real evaluation of a process or process improvement proposal is to have people using it, since the process is just a description until it is used by people. Empirical studies are crucial to the evaluation of processes and human-based activities. It is also beneficial to use empirical studies when there is a need to evaluate the use of software products or tools. Experimentation provides a systematic, disciplined, quantifiable and controlled way of evaluating human-based activities. This is one of the main reasons why empirical research is common in social and behavioral sciences, see for example [Robson93].

In addition, empirical studies and experiments in particular are also important for researchers in software engineering. New methods, techniques, languages and tools should not just be suggested, published and marketed. It is crucial to evaluate new inventions and proposals in comparison with existing ones. Experimentation provides this opportunity, and should be used accordingly. In other words, we should use the methods and strategies available when conducting research in software engineering. This is further discussed next.

1.2 Science and software engineering

Software engineering is a cross-disciplinary subject. It stretches from technical issues such as databases and operating systems, through language issues, for example, syntax and semantics, to social issues and psychology. Software development is human-intensive; we are, at least today, unable to manufacture new software. It is a discipline based on creativity and the ingenuity of the people working in the field. Nevertheless, we should, when studying and doing research in software engineering, aim at treating it as a science. This implies using scientific methods for doing research and when making decisions regarding changes in the way we develop software.

In order to perform scientific research in software engineering, we have to understand the methods that are available to us, their limitations and when they can be applied. Software engineering stems from the technical community. Thus, it is natural to look at the methods used for research in, for example, hardware design and coding theory, but based on the nature of software engineering we should look at other disciplines too.

In [Glass94], four research methods in the field of software engineering are summarized. They were initially presented in a software engineering context in [Basili93]. The methods are:

• The scientific method. The world is observed and a model is built based on the observation, for example, a simulation model.

• The engineering method. The current solutions are studied and changes are proposed, and then evaluated.

• The empirical method. A model is proposed and evaluated through empirical studies, for example, case studies or experiments.

• The analytical method. A formal theory is proposed and then compared with empirical observations.

The engineering method and the empirical method can be seen as variations of the scientific method [Basili93].

Traditionally, the analytical method is used in the more formal areas of electrical engineering and computer science. The scientific method is used in applied areas, such as simulating a telecommunication network in order to evaluate its performance. It should, however, be noted that simulation as such is not only applied in the scientific method. Simulation may be used as a means for conducting an experiment as well. The engineering method is probably dominating in industry.

The empirical studies have traditionally been used in social sciences and psychology, where we are unable to state any laws of nature, as in physics. In social sciences and psychology, they are concerned with human behavior. The important observation, in this context, is hence that software engineering is very much governed by human behavior through the people developing software. Thus, we cannot expect to find any formal rules or laws in software engineering except perhaps when focusing on specific technical aspects. The focus of this book is on applying and using empirical studies in software engineering. The objective is in particular to emphasize the underlying process when performing empirical studies in general and experimentation in particular. An experiment process is presented, which highlights the basic steps to perform experiments, provides guidelines of what to do and exemplifies the steps using software engineering examples.

It must be noted that it is not claimed that the analytical, scientific and engineering methods are inappropriate for software engineering. They are necessary for some parts of software engineering, for example, we may build mathematical models for software reliability growth [Lyu96]. Moreover, the research methods are not orthogonal, and hence it may, for example, be appropriate to conduct an empirical study within, for example, the engineering method. The important point is that we should make better use of the methods available within empirical research. They are frequently used in other disciplines, for example, behavioral sciences, and the nature of software engineering has much in common with disciplines outside the technical parts of engineering.

The need for experimentation in software engineering was really emphasized for the first time in the middle of the 1980's in [Basili86]. Other articles stressing the need for empiricism in software engineering have since been published, see for example [Basili96b, Fenton94a, Glass94, Potts93, Kitchenham95, Tichy98]. The lack of empirical evidence in software engineering research is stressed in [Tichy95, Zelkowitz98]. The latter publications indicate that the research in software engineering is still too much of advocacy research [Potts93]. A more scientific approach to software engineering is needed.

The focus of this book is on software engineering and the application and use of empirical studies, in particular experimentation, in software engineering. Empirical strategies in software engineering include:

• setting up formal experiments,

• studying real projects in industry, i.e. performing a case study, and

• performing surveys through, for example, interviews.

These strategies are described in some more detail in Chapter 2 before focusing on experimentation. A more general introduction to these research strategies can be found in, for example, [Robson93]. The strategies are neither completely orthogonal nor competing. They provide a convenient classification, but some studies may be viewed as combinations of them or somewhere between two of them. Thus, there are both similarities and differences between the strategies.

The main reason to use experimentation in software engineering is to enable understanding and identification of relationships between different variables. A number of preconceived ideas exist, but are they true? Does object-orientation improve reuse? Are inspections cost-effective? Should we have inspection meetings or is it sufficient to hand-in the remarks to a moderator? This type of questions can be investigated in order to improve our understanding of software engineering. Improved understanding is the basis for changing and improving the way we work, hence empirical studies in general and experimentation in particular are important.

The introduction to the area is based on the introduction of a process for experimentation. The basic steps in the process can be used for other types of empirical studies too. The focus is, however, on providing guidelines and support for performing experiments in software engineering. Furthermore, it should be noted that "true" experiments, i.e. experiments with full randomization, are difficult to perform in software engineering. Software engineering experiments are often quasiexperiments, i.e. experiment in which it, for example, has not been possible to select participants in the experiments by random. Quasi-experiments are important, and they can provide valuable results. The process presented is aimed at both "true" experiments and quasi-experiments. The latter is particularly supported by a thorough discussion of threats to experiments.

Thus, the intention of this book is to provide an introduction to empirical studies and experimentation, in order to highlight the opportunities and benefits of doing experiments in the field of software engineering. The empirical research method can, and should be used more in software engineering. The arguments against empirical studies in software engineering are refuted in [Tichy98]. Hopefully, this introduction to experimentation in software engineering facilitates the use of empirical studies and experimentation both within software engineering research and practice.

Table of Contents

Foreword. Preface. Acknowledgment. 1. Introduction. 2. Empirical strategies. 3. Measurement. 4. Experiment process. 5. Definition. 6. Planning. 7. Operation. 8. Analysis and interpretation. 9. Presentation and package. 10. Literature survey. 11. Example: Experiment process. 12. Example: C versus C++. 13. Exercises. Appendix A: Statistical tables. Appendix B: Experiment process overview. References. About the authors. Index.

Editorial Reviews

‘This is a well written and concise book, which provides the reader with the essentials necessary to design, conduct and analyse a software engineering experiment. Its value lies in that it is specifically written for the software engineering field, and has surveyed the major contributions by a number of leading researchers in this area. A ‘how to' book is always welcome, both as a useful starting point to the inexperienced and as a helpful reference and reminder of best practice to others.'
Software Testing Verification and Reliability, 11 (2001)

From the Publisher