Read an Excerpt

CHAPTER 1

The Evolution of the Citation Project

Developing a Pilot Study from Local to Translocal

Sandra Jamieson

ABSTRACT

The historical narrative in this chapter traces the evolution of the Citation Project from its origins in a graduate seminar to the publication of pilot data (Howard, Serviss, and Rodrigue 2010) and the development of a transcontextual, multisite research project with internationally reported and replicated data. Based on interviews with principal and participating researchers and coders, analysis of research and coding notebooks, two blogs and various shared Google Docs, and e-mails as well as shared personal experiences, this chapter offers a historical account of methodological development that reveals the complexity and messiness of multisite research as well as the necessary adjustments that allow pilot research to be scaled to multisite projects. By being willing to expose not only their methods but also the false starts, challenges, and lessons they learned, Citation Project researchers hope to ease the transition to data-driven research and thereby increase the frequency of information-based policies and pedagogies.

INTRODUCTION

This chapter provides an antidote to the (necessarily) highly systematized accounts of research processes to which new researchers frequently turn, accounts that in users' minds too easily become ideals to be achieved and standards by which to measure their work. Books such as Johnny Saldaña's (2013), Stefan Titscher et al.'s (2000), and John Creswell's (2014) are invaluable procedural guides for conducting research — and highly recommended — but while they do acknowledge the unruliness of qualitative research, they nevertheless present a linear, cleaned-up version of the process that can leave new researchers at a loss when their own work is stalled. Along with recent calls for writing studies researchers to share their methods and research design (Lunsford 2013), there is also a need for transparency in our field's research narratives. The reality of research, especially data-driven research, is that it is often a very messy, start-and-stop, revise-and-start-over process marked by frustration at many points along the way, as Rebecca Moore Howard and I noted in a keynote to the CCCC Research Network Forum (Howard and Jamieson 2012). Those of us trained in literary or rhetorical research methods are generally ill prepared for the challenges and time-consuming nature of datadriven research, and because it has not been a staple of our field until very recently, many of us lack mentors who can help. Similarly, most of us are unused to working collaboratively on research and writing, something probably essential for largerscale research as our colleagues in the social and natural sciences learned long ago. There are many things to consider before beginning a RAD research project; this chapter presents some of those factors in hopes of encouraging other such endeavors.

Collaborative RAD research is infinitely more rewarding than anyone imagines, though, and, as the other chapters in this book reveal, has the potential to lead to the kinds of changes in pedagogies, policies, and practices many of us desire. I believe research narratives that are honest about failures and setbacks, coupled with the methods and design of the final research projects they engendered, will help researchers — experienced and prospective alike — imagine and plan large-scale research projects of their own. I hope narratives like this one will also help my fellow researchers work through the inevitable messiness and rethinking that brings such projects to successful completion.

The research project that is the focus of this chapter is the Citation Project, specifically a study of eight hundred pages of researched writing produced by 174 students enrolled in first-year writing courses at sixteen US colleges and universities. Researchers coded both the kinds of sources selected and the ways students incorporated information from those sources into their papers (summary, paraphrase, quotation, patchwriting, or copying). They also coded the kinds of sources used, including type, length, and reading difficulty.

The methods and findings of the Citation project sixteen-school study have been described elsewhere (Jamieson 2013; Jamieson and Howard 2013), and documents from that research are included in the appendix to this chapter. My purpose here is not to describe those methods per se or discuss the findings (although I will mention them by way of comparison) but to narrate the evolution of the project's procedures and coding methods over a considerable time and through a series of messy drafts that ultimately allowed the collection and analysis of transcontextual RAD data on a broad scale. Using information from interviews with founding researchers (principal and participating researchers and coders), analysis of research and coding notebooks, two blogs, and various shared Google Docs and e-mails, in addition to personal experience, I will describe the various challenges encountered as the research moved from a series of questions generated in a graduate seminar to a single-institution study, then to a three-school study conducted after I became one of the two principal researchers, and thence to the sixteen-school study whose data I reported above. By sharing not only their methods but also the false starts, challenges, and lessons they learned, Citation Project researchers hope to ease the transition to data-driven research and thereby increase the frequency of information-based policies and pedagogies.

While there are things we all wish we knew before we started this research, what is more useful to future researchers is what we learned in the process and the ways it led us to refine our research and develop methods we can share with others who are conducting their own citation context research or replicating the sixteen-school study.

CITATION PROJECT ORIGINS

It is instructive to trace the development of large-scale research like this to the various points of origin, both to give credit to the many people involved and to emphasize the importance of ideological and theoretical frameworks that necessarily shape research. Going back to the original motives and influences can help researchers when they become blocked, reminding them of the reasons they are doing what they are doing and how many problems they have already overcome — which is why the kind of record keeping that led to this chapter is so important.

Origin Stories

Everything has an origin story; the Citation Project has two.

Narrative 1: The Linear Narrative

Rebecca Moore Howard first became interested in student source use in the mid-1980s, coining the term patchwriting in 1993 when she described her analysis of writing by students in one of her classes (Howard 1993). A decade later, Diane Pecorari set out to empirically test Howard's claims in the writing of second-language graduate students (Pecorari 2003). These two articles helped shape a doctoral seminar in curriculum design focused on authorship studies at Syracuse University in the fall of 2006, from which developed a small class project that experimented with textual coding and then developed into an actual pilot study at a single institution. The results of that study were published in 2010 (Howard, Serviss, and Rodrigue).

I joined the project in 2008. While Serviss and Rodrigue turned their attention to other projects, Howard and I expanded the single-institution pilot study to three institutions (a liberal arts college, a private research university, and a state university). We brought in contributing researchers to help test the citation-analysis methods and code the papers, and we described our findings in a presentation entitled "The Citation Project Three-School Study" at the Conference on College Composition and Communication (Benedicks et al. 2010; Jamieson 2010).

After that three-school study came a plan to collect and code papers from ten colleges and universities that would represent a wide geographic distribution and institutional variety. Ultimately, 174 papers from sixteen institutions were collected and coded and the initial data from all sixteen was presented at the CCCC conference two years later (Jamieson 2012). Although Citation Project researchers generally refer to the initial single-institution study as the pilot study in that it developed the general coding categories used to code source use throughout the expansion of the project, the coding procedures and terminology continued to be refined in the process of conducting the study of papers from three quite different institutions, and those papers were ultimately recoded using the final criteria developed for the sixteen-school study. New research is now being conducted by scholars involved in that initial study, and new projects are developing in the United States and abroad, necessitating more fine tuning of methods to accommodate different citation styles. In 2012, principal researchers Rebecca Moore Howard and Sandra Jamieson were joined by Tricia Serviss, a co-researcher for the single-school pilot study and coauthor of this collection. Serviss and I are also working with Angela Feekery to revise and repeat the study across universities in her native New Zealand.

Narrative 2: Theoretical Underpinnings

There is also a second origin story. This one concerns not the Citation Project research per se but the theoretical and methodological frameworks that made it not only possible for Howard and I to imagine doing such research but impossible for us not to do so. And the origin story here is the moment when we were confronted by the realization that our field needs data-driven research and we needed to do it. This realization led me to take a statistics class at my institution and therefore to be able to talk about the language of statistical analysis when I read a draft in progress of the article "Writing from Sources" (Howard, Serviss, and Rodrigue 2010); it also led me ultimately to become a principal researcher with the Citation Project.

The realization about the importance of data-driven research came as we listened to Chris Anson's keynote presentation at the 2006 conference of the Council of Writing Program Administrators (Anson 2006; 2008). In that keynote, Anson argued that the shared assumptions of writing studies researchers and practitioners make it relatively easy to make a case for change others would endorse. Outside our own field, though, those shared assumptions do not necessarily prevail, and the ethos of the person making the argument is a much less powerful piece of evidence than it is within our field. Anson contended that if writing program administrators are to persuade cross-curricular colleagues, higher administration, fund-granting foundations, legislators, and the like to do or change anything, those WPAs need data-based evidence. Statistics, he specified, are the gold standard of universal evidence.

Rebecca Moore Howard and I were sitting side by side in the audience in Chattanooga when he delivered that keynote, and we spent the remainder of the conference talking about its possible implications for our own work. We two had been coauthors since 1993, when we began The Bedford Guide to Teaching Writing in the Disciplines (Howard and Jamieson 1995), and we shared scholarly and pedagogical interests. This collaborative history is important because it allowed us to work through the inevitable disagreements and setbacks as we developed and worked on the Citation Project. However, what has really motivated us through this work and continues to motivate us as we write and speak about the data and expand the project is the idea that statistical and transcontextual evidence can bring change in ways anecdotal evidence cannot (see Howard 2011, 2014). Before Howard published her data about the students in her class in 1993, other scholars had published individual case studies highlighting the challenges experienced by developmental writers as they try to incorporate sources (Hull and Rose 1989 and 1990, for example) and try to build papers from those sources (Kantz 1990; Kennedy 1985), but the numbers provided in Howard's 1993 article had resonance and, of course, she also named the phenomenon. Once named and defined, patchwriting could be identified as such and measured, and scholars could begin the process of establishing it as a separate category from plagiarism (Jamieson 2016). The frequency with which the 2010 article describing Howard, Serviss, and Rodrigue's initial study is cited speaks to a larger thirst for data-driven findings. It is the need for more data that motivated Howard and me because we both want to see the pedagogical and policy changes Anson argues, and we agree, only data can bring.

DESIGNING CITATION CONTEXT RESEARCH

Howard shared a copy of Anson's keynote with students in her doctoral seminar in fall 2006 along with the recommendations of the CCCC Caucus on Intellectual Property. In response, the students — Sarah Etlinger, Tanya Rodrigue, Tricia Serviss, Zosha Stuckey, and Terri White — set about exploring how such data-driven research could be used to investigate authorship issues (the topic of the class). They focused on how to study what undergraduates do with the sources they use in their college papers and collectively designed (and named) the research project described in the class blog (Figure 1.1) and drafted information for an IRB application.

The "pre-determined units of analysis" (Figure 1.1) were those developed by Diane Pecorari (2003) — transparent and opaque source use — to test Howard's hypotheses from ten years earlier (Howard 1993). Working with nonnative speakers of English, Pecorari took ten randomly selected pages from portions of seventeen draft dissertations and collected an additional ten pages from each of eight published PhD theses. Samples were "divided into passages of varying length, the passage boundaries being determined by the source use" (Pecorari 2003, 322), allowing her to compare each passage with the source from which it drew. She coded source use as transparent or opaque. If one could tell where a source was used (ideally where it began and ended), the citation was transparent; where it was difficult to separate the source from the student prose, the material was coded as opaque. Working from this same method of citation context analysis, the seminar participants set out to replicate Pecorari's research by coding five student papers from their own institution. While no coded material was preserved, the blog contains discussions of individual sections and the struggles the researchers experienced as they tried to code ("CCR732F06" 2006). Without definition, the codes were difficult to consistently apply and the coders simply couldn't find consensus. By the time they had tried to code three papers, they had rejected the terms transparent and opaque as insufficient. They also concluded that to understand how the sources were being used, one would need to read every cited source, something they had been unable to do and, they worried, an unsustainable method. So here in this early modification of both coding method and practice are two important realizations that helped shape Citation Project research: all cited sources must be read by coders; all coding categories must be clearly defined.

The goal defined in this blog post — to develop "responsible research which will help promote productive pedagogy and aid in a better comprehension of student work when engaging sources"— remains the goal of the Citation Project today, and the process they describe can also be traced in current methods. Ultimately, although the class did not complete the research they designed, Citation Project research did follow the trajectory described in Figure 1.1 and "develop[ed] into a national endeavor on a variety of campuses." The process of moving from initial idea to action was not the smooth linear process many people imagine when they set out to do research — or when they hear single-point origin stores. Many theoretical discussions, microstudies, revisions, and collaborations go into the design and execution of successful transcontextual research, and frequently people leave and others join along the way. Such fluidity is alien to writing studies research but common in other fields where transcontextual research is the norm.

FROM DESIGN TO PILOT STUDY

The conclusion of the seminar participants that researchers needed to move from overly narrow coding categories to broader and more focused categories is typical in the development of coding projects (Saldaña 2013, 11). It is not surprising, then, that at the conclusion of the semester when Howard, Serviss, and Rodrigue decided to revise the initial ideas explored in the seminar into a pilot study, their first step was to work on coding categories. The three researchers spent a semester designing coding categories, developing the research plan, securing IRB clearance at their institution, and gathering student papers produced in a required research-writing course. In this first pilot phase, Howard, Serviss, and Rodrigue coded eighteen student papers that had been submitted for a grade in fifteen sections of a composition course at a private, not-for-profit university with an RU/H Carnegie basic classification (Howard, Serviss, and Rodrigue 2010).

(Continues…)

---

Excerpted from "Points of Departure"
by .
Copyright © 2017 University Press of Colorado.
Excerpted by permission of University Press of Colorado.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

---