Data Quality Assessment (Data Quality for Practitioners Series)

by Arkady Maydanchik

Overview

Imagine a group of prehistoric hunters armed with stone-tipped spears. Their primitive weapons made hunting large animals, such as mammoths, dangerous work. Over time, however, a new breed of hunters developed. They would stretch the skin of a previously killed mammoth on the wall and throw their spears, while observing which spear, thrown from which angle and distance, penetrated the skin the best. The data gathered helped them make better spears and develop better hunting strategies.

Quality data is the key to any advancement, whether it’s from the Stone Age to the Bronze Age. Or from the Information Age to whatever Age comes next. The success of corporations and government institutions largely depends on the efficiency with which they can collect, organize, and utilize data about products, customers, competitors, and employees. Fortunately, improving your data quality doesn’t have to be such a mammoth task.

DATA QUALITY ASSESSMENT is a must read for anyone who needs to understand, correct, or prevent data quality issues in their organization. Skipping theory and focusing purely on what is practical and what works, this text contains a proven approach to identifying, warehousing, and analyzing data errors – the first step in any data quality program. Master techniques in:

• Data profiling and gathering metadata
• Identifying, designing, and implementing data quality rules
• Organizing rule and error catalogues
• Ensuring accuracy and completeness of the data quality assessment
• Constructing the dimensional data quality scorecard
• Executing a recurrent data quality assessment

Product Details

ISBN-13:	9780977140022
Publisher:	Technics Publications, LLC
Publication date:	05/28/2007
Series:	Data Quality for Practitioners Series
Edition description:	TECHNICS PUBLICATIONS LLC
Pages:	321
Product dimensions:	6.90(w) x 9.90(h) x 0.90(d)

Table of Contents

Chapter 2 Data Quality Program Overview 23
2.1. Data Quality Assessment 25
2.2. Data Cleansing 29
2.3. Monitoring Data Integration Interfaces 31
2.4. Ensuring Data Quality in Data Conversion and Consolidation 33
2.5. Building Data Quality Meta Data Warehouse 37
Summary 38
Chapter 3 Data Quality Assessment Overview 39
3.1. Project Team 41
3.2. Project Plan Overview 42
3.3. Planning Phase 44
3.4. Preparation Phase 46
3.5. Implementation Phase 48
3.6. Fine-Tuning Phase 52
3.7. Ongoing Data Quality Monitoring 54
Summary 55
PART II - Data Quality Rules 57
Chapter 4 Attribute Domain Constraints 63
4.1. Introduction to Attribute Domain Constraints 64
4.2. Attribute Profiling 65
4.3. Optionality Constraints 66
4.4. Attribute Format Constraints 69
4.5. Valid Value Constraints 71
4.6. Precision Constraints 74
Summary 77
Chapter 5 Relational Integrity Rules 79
5.1. Relational Data Model Basics 80
5.2. Identity Rules 82
5.3. Reference Rules 85
5.4. Cardinal Rules 86
5.5. Inheritance Rules 91
Summary 92
Chapter 6 Rules for Historical Data 93
6.1. Introduction to Historical Data 94
6.2. Basic Data Quality Rules for Historical Data 98
6.3. Advanced Data Quality Rules for Historical Data 102
6.4. Data Quality Rules for Event Histories 106
Summary 111
Chapter 7 Rules for State-Dependent Objects 113
7.1. Introduction to State-Dependent Objects 114
7.2. Identifying State-Dependent Entities 117
7.3. Profiling State-Transition Models 119
7.4. Rules Derived from State-Transition Diagrams 124
7.5. Timeline Constraints 128
7.6. Advanced Rules 133
Summary 141
Chapter 8 Attribute Dependency Rules 143
8.1. Introduction to Attribute Dependency Rules 144
8.2. Identifying Dependencies through Analysis 151
8.3. Identifying Dependencies through Data Profiling 156
8.4. Identifying Dependencies Across Data Sources 162
Summary 167
PART III - Data Quality Assessment 169
Chapter 9 Implementing Data Quality Rules 173
9.1. Project Scope and Rule Design 174
9.2. Selecting Optimal Rule Design 180
9.3. Rule Cataloguing 188
9.4. Rule Coding 193
Summary 198
Chapter 10 Fine-Tuning Data Quality Rules 201
10.1. Rule Imperfections 202
10.2. Rule Fine-Tuning Process 205
10.3. Identifying Rule Imperfections 206
10.4. Analyzing Imperfection Patterns 211
10.5. Eliminating False Positives 213
10.6. Handling False Negatives 216
10.7. Handling Uncertainty in Error Location 218
Summary 221
Chapter 11 Cataloguing Errors 223
11.1. Error Catalogue Basics 224
11.2. Recording Missing Records 226
11.3. Errors Affecting Multiple Records 228
11.4. Error Groups 231
11.5. Subject-Level Error Tracking 234
11.6. Error Messages 238
Summary 240
Chapter 12 Measuring Data Quality Scores 243
12.1. Introduction to Aggregate Scores 244
12.2. Score Tabulation Process Overview 248
12.3. Building Score Catalogue 249
12.4. Tabulating Record-Level Scores 255
12.5. Adjusting Scores for Rule Imperfections 261
12.6. Tabulating Subject-Level Scores 266
Summary 270
Chapter 13 Data Quality Meta Data Warehouse 273
13.1. Data Quality Assessment Meta Data 274
13.2. Data Quality Scorecard 285
13.3. Other DQMDW Functions and Reports 292
Summary 294
Chapter 14 Recurrent Data Quality Assessment 297
14.1. Basics of Recurrent Data Quality Assessment 298
14.2. Data Quality Changes on Atomic Level 301
14.3. Adding Time Dimension to DQMDW 303
14.4. Executing Assessment Runs Against Production Data 306