Read an Excerpt
1 Introduction
The information technology revolution has meant, among other things, that software has become a part of more and more products. Software is found in products ranging from toasters to space shuttles. This means that a vast amount of software has been and is being developed. Software development is by no means easy; it is a highly creative process. The rapid growth of the area has also meant that numerous software projects have run into problems in terms of missing functionality, cost overruns, missed deadlines and poor quality. These problems or challenges were identified already in the 1960's, and in 1968 the term "software engineering" was coined with the intention of creating an engineering discipline that focused on the development of software systems.
Software engineering is formally defined by IEEE [IEEE90] as "software engineering means application of a systematic, disciplined, quantifiable approach to development, operation and maintenance of software". Software engineering in general is presented and discussed in books as, for example, [Sommerville95, Pfleeger98]. The objective here is to present how empirical studies and experiments in particular fit into a software engineering context. Three aspects in the definition above are of particular importance here. First, it implies a software process through pointing at different life cycle phases; secondly, it stresses the need for a systematic and disciplined approach; finally, it highlights the importance of quantification. The use of empirical studies is related to all three of them. The software engineering context is further discussed in Section 1.1. The need to turn software engineering more scientific and how empirical studies play an important role in this is discussed in Section 1.2.
1.1 Software engineering context
A software process model is used to describe the steps to take and the activities to perform when developing software. Examples of software process models are the waterfall model, incremental development, evolutionary development and the spiral model. These and other models are discussed in the general software engineering literature. A simplistic view of the software process is shown in Figure 2. It should be noted that the process is crucial whether we work with development of a new product or maintenance of an existing product.
In Figure 2, an idea and resources, primarily in the form of people, are inputs to the software process, and the people develop a software product going through the different steps and performing the different activities in the software process.
The development of software products is many times a complex task. Software projects may run over a long period of time and involve many people, due to the complexity of the software products that are developed. This implies that the software process often also becomes very complex. It consists of many different activities and many documents are written before the final product can be delivered. The complexity of the software process means that it is difficult to optimize it or even find a good enough process. Thus, it is important for companies to strive to improve their way of making business if they intend to stay competitive. This means that most companies are continuously trying to improve their software process in order to improve the products, lower the cost and so forth. The software process stresses the need for a systematic and disciplined approach of working. This approach is also needed when improving the software process, and hence a process improvement process is needed.
An example of an improvement process tailored for software development is the Quality Improvement Paradigm (QIP) [Basili85]. It consists of several steps to support a systematic and disciplined approach to improvement. The QIP is presented briefly in Section 2.6.2. A more general improvement process is the well-known Plan/Do/Check/Act cycle [Demming86]. The improvement processes include two activities, although the same terminology is not always used, that we would like to highlight:
- Assessment of the software process.
- Evaluation of a software process improvement proposal.
The assessment is conducted to identify suitable areas for improvement. Several models exist for assessing the software process. The most well known is probably the Capability Maturity Model (CMM) from Software Engineering Institute at Carnegie-Mellon University, USA [Paulk93]. The assessment models help in pinpointing where improvements are needed. The CMM has five maturity levels with so called key process areas on each level. It is recommended that companies focus on improvement areas according to their maturity level.
Assuming that it is possible to identify areas for improvement through some form of assessment, the next step is to determine how these areas of improvement may be addressed to cope with the identified problems. For example, if too many defects are found in system testing, it may be possible to improve earlier testing, inspections or even specific parts in the development, for example, software design. The objective is that the assessment of the current situation and knowledge about the state-of-the-art should result in that concrete process improvement proposals can be identified. When the improvement proposals have been identified, it is necessary to determine which to introduce, if any. It is often not possible just to change the existing software process without having more information about the actual effect of the improvement proposal. In other words, it is necessary to evaluate the proposals before making any major changes.
One problem that arises is that a process improvement proposal is very hard to evaluate without human involvement directly. For a product, it is possible to first build a prototype to evaluate if it is something to work further with. For a process, it is not possible to build a prototype. It is possible to make simulations and compare different processes, but it should be remembered that this is still an evaluation that is based on a model. The only real evaluation of a process or process improvement proposal is to have people using it, since the process is just a description until it is used by people. Empirical studies are crucial to the evaluation of processes and human-based activities. It is also beneficial to use empirical studies when there is a need to evaluate the use of software products or tools. Experimentation provides a systematic, disciplined, quantifiable and controlled way of evaluating human-based activities. This is one of the main reasons why empirical research is common in social and behavioral sciences, see for example [Robson93].
In addition, empirical studies and experiments in particular are also important for researchers in software engineering. New methods, techniques, languages and tools should not just be suggested, published and marketed. It is crucial to evaluate new inventions and proposals in comparison with existing ones. Experimentation provides this opportunity, and should be used accordingly. In other words, we should use the methods and strategies available when conducting research in software engineering. This is further discussed next.
1.2 Science and software engineering
Software engineering is a cross-disciplinary subject. It stretches from technical issues such as databases and operating systems, through language issues, for example, syntax and semantics, to social issues and psychology. Software development is human-intensive; we are, at least today, unable to manufacture new software. It is a discipline based on creativity and the ingenuity of the people working in the field. Nevertheless, we should, when studying and doing research in software engineering, aim at treating it as a science. This implies using scientific methods for doing research and when making decisions regarding changes in the way we develop software.
In order to perform scientific research in software engineering, we have to understand the methods that are available to us, their limitations and when they can be applied. Software engineering stems from the technical community. Thus, it is natural to look at the methods used for research in, for example, hardware design and coding theory, but based on the nature of software engineering we should look at other disciplines too.
In [Glass94], four research methods in the field of software engineering are summarized. They were initially presented in a software engineering context in [Basili93]. The methods are:
- The scientific method. The world is observed and a model is built based on the observation, for example, a simulation model.
- The engineering method. The current solutions are studied and changes are proposed, and then evaluated.
- The empirical method. A model is proposed and evaluated through empirical studies, for example, case studies or experiments.
- The analytical method. A formal theory is proposed and then compared with empirical observations.
The engineering method and the empirical method can be seen as variations of the scientific method [Basili93].
Traditionally, the analytical method is used in the more formal areas of electrical engineering and computer science. The scientific method is used in applied areas, such as simulating a telecommunication network in order to evaluate its performance. It should, however, be noted that simulation as such is not only applied in the scientific method. Simulation may be used as a means for conducting an experiment as well. The engineering method is probably dominating in industry.
The empirical studies have traditionally been used in social sciences and psychology, where we are unable to state any laws of nature, as in physics. In social sciences and psychology, they are concerned with human behavior. The important observation, in this context, is hence that software engineering is very much governed by human behavior through the people developing software. Thus, we cannot expect to find any formal rules or laws in software engineering except perhaps when focusing on specific technical aspects. The focus of this book is on applying and using empirical studies in software engineering. The objective is in particular to emphasize the underlying process when performing empirical studies in general and experimentation in particular. An experiment process is presented, which highlights the basic steps to perform experiments, provides guidelines of what to do and exemplifies the steps using software engineering examples.
It must be noted that it is not claimed that the analytical, scientific and engineering methods are inappropriate for software engineering. They are necessary for some parts of software engineering, for example, we may build mathematical models for software reliability growth [Lyu96]. Moreover, the research methods are not orthogonal, and hence it may, for example, be appropriate to conduct an empirical study within, for example, the engineering method. The important point is that we should make better use of the methods available within empirical research. They are frequently used in other disciplines, for example, behavioral sciences, and the nature of software engineering has much in common with disciplines outside the technical parts of engineering.
The need for experimentation in software engineering was really emphasized for the first time in the middle of the 1980's in [Basili86]. Other articles stressing the need for empiricism in software engineering have since been published, see for example [Basili96b, Fenton94a, Glass94, Potts93, Kitchenham95, Tichy98]. The lack of empirical evidence in software engineering research is stressed in [Tichy95, Zelkowitz98]. The latter publications indicate that the research in software engineering is still too much of advocacy research [Potts93]. A more scientific approach to software engineering is needed.
The focus of this book is on software engineering and the application and use of empirical studies, in particular experimentation, in software engineering. Empirical strategies in software engineering include:
- setting up formal experiments,
- studying real projects in industry, i.e. performing a case study, and
- performing surveys through, for example, interviews.
These strategies are described in some more detail in Chapter 2 before focusing on experimentation. A more general introduction to these research strategies can be found in, for example, [Robson93]. The strategies are neither completely orthogonal nor competing. They provide a convenient classification, but some studies may be viewed as combinations of them or somewhere between two of them. Thus, there are both similarities and differences between the strategies.
The main reason to use experimentation in software engineering is to enable understanding and identification of relationships between different variables. A number of preconceived ideas exist, but are they true? Does object-orientation improve reuse? Are inspections cost-effective? Should we have inspection meetings or is it sufficient to hand-in the remarks to a moderator? This type of questions can be investigated in order to improve our understanding of software engineering. Improved understanding is the basis for changing and improving the way we work, hence empirical studies in general and experimentation in particular are important.
The introduction to the area is based on the introduction of a process for experimentation. The basic steps in the process can be used for other types of empirical studies too. The focus is, however, on providing guidelines and support for performing experiments in software engineering. Furthermore, it should be noted that "true" experiments, i.e. experiments with full randomization, are difficult to perform in software engineering. Software engineering experiments are often quasiexperiments, i.e. experiment in which it, for example, has not been possible to select participants in the experiments by random. Quasi-experiments are important, and they can provide valuable results. The process presented is aimed at both "true" experiments and quasi-experiments. The latter is particularly supported by a thorough discussion of threats to experiments.
Thus, the intention of this book is to provide an introduction to empirical studies and experimentation, in order to highlight the opportunities and benefits of doing experiments in the field of software engineering. The empirical research method can, and should be used more in software engineering. The arguments against empirical studies in software engineering are refuted in [Tichy98]. Hopefully, this introduction to experimentation in software engineering facilitates the use of empirical studies and experimentation both within software engineering research and practice.
