Kathleen Blake Yancey - "Historicizing Writing Assessment"
Yancey takes readers through the development of written assessment, dividing the chronology into three stages: 1950's - 1970, 1970-1986, 1986 - present.
In the first stage, writing assessment for placement and program review was completed through the use of multiple choice reading and editing tests, which while creating a high degree 0f reliability, were, rightly, of questionable validity. The editing test is an indirect measure of writing ability, although Paul Diederich, an ETS researcher, during the 50's, characterized the correlation between a good reading test and a good teacher's estimate of their own student's writing ability at .65. Diederich characteriZed a good objective test at .60 with such judgements, and, actually, found a single two-hour essay, graded twice by expert college board readers at .45 to .50.
In this case, the actual "essay" test fares poorly in contrast to the other supposed measure of writing ability. This is, however, probably indicative of problems with test design and scoring rather than an indictment of using WRITING to assess writing, which is, obviously, the most reliable (testing what it purports to measure) method/
With the second wave, rather than focusing on reliability, which the multiple choice tests were excellent for, the testers, administration and faculty switched their focus to establishing validity with written assessment. The result of this push was holistic scored essay exams. This push was spearheaded by Edward White in his position as the director of the California State University Freshman English Equivalency Examination program.
In 1971, the CSU chancellor adopted a plan to essentially eliminate the first-year of college by replacing it with standardized test which, if a student passed. would replace those required courses. Faculty resisted vehemently because of what such a move said about those courses they were teaching, e.g., that they were largely remedial and unnecessary for the most part, for many students. It was due to this impetus that White and others organized, and rather than defeat the measure outright, compromised (wisely) with the chancellor and agreed to take the issue of equivalency up themselves, devising their own means for establishing equivalency -- the holistically scored essay test.
In this case, reliability was established by adhereing to stringent procedures:
1. using writing prompts that directed students;
2. selecting "achor" papers and scoring guides that directed teacher-readers who rated; and
3. devising methods of calculating "acceptable agreement. (Yancey 490).
The third wave of writing assessment developed out of the second waves focus and need to establish validity: the writing portfolio. Gordon Brossell summarizes:
we know that for a valid test of writing performance, multiple writing samples written on different occassions and in various rhetorical modes are preferable to single document drawn from an isolated writing instance (qtd in Yancey 491).
This method, collecting a number of pieces written in real rhetorical situations, rated by teachers of writing as either pass / fail, rather than the holistic scale of, for example, 1-6, improved the reliability of the testing by redefining the term within this context. According to Yancey, the raters "were not trained to agree, as in the holistic scoring model, but rather released to read, to negotiate among themselves, 'hammering out an agreeable compromise.' (492).
In this way reliability is increased through the negotiation of what Peter Elbow terms "community standards" (Elbow, Portfolios, xxi). Throughout the development of writing assessment, classroom teachers had lamented the fact that what was done with assessment did not relate to what they were doing in their classes creating dissonance between testing and practice. Finally, with Elbow and Belanoff and others developing the portfolio as a model for placement and program assessment, as well as classroom assessment, the two were merged: knowledge is created about assessment, but also about, and informing practice as well.
At this point, I might introduce information from SWIC's basic writing assessment program.
In the first stage, writing assessment for placement and program review was completed through the use of multiple choice reading and editing tests, which while creating a high degree 0f reliability, were, rightly, of questionable validity. The editing test is an indirect measure of writing ability, although Paul Diederich, an ETS researcher, during the 50's, characterized the correlation between a good reading test and a good teacher's estimate of their own student's writing ability at .65. Diederich characteriZed a good objective test at .60 with such judgements, and, actually, found a single two-hour essay, graded twice by expert college board readers at .45 to .50.
In this case, the actual "essay" test fares poorly in contrast to the other supposed measure of writing ability. This is, however, probably indicative of problems with test design and scoring rather than an indictment of using WRITING to assess writing, which is, obviously, the most reliable (testing what it purports to measure) method/
With the second wave, rather than focusing on reliability, which the multiple choice tests were excellent for, the testers, administration and faculty switched their focus to establishing validity with written assessment. The result of this push was holistic scored essay exams. This push was spearheaded by Edward White in his position as the director of the California State University Freshman English Equivalency Examination program.
In 1971, the CSU chancellor adopted a plan to essentially eliminate the first-year of college by replacing it with standardized test which, if a student passed. would replace those required courses. Faculty resisted vehemently because of what such a move said about those courses they were teaching, e.g., that they were largely remedial and unnecessary for the most part, for many students. It was due to this impetus that White and others organized, and rather than defeat the measure outright, compromised (wisely) with the chancellor and agreed to take the issue of equivalency up themselves, devising their own means for establishing equivalency -- the holistically scored essay test.
In this case, reliability was established by adhereing to stringent procedures:
1. using writing prompts that directed students;
2. selecting "achor" papers and scoring guides that directed teacher-readers who rated; and
3. devising methods of calculating "acceptable agreement. (Yancey 490).
The third wave of writing assessment developed out of the second waves focus and need to establish validity: the writing portfolio. Gordon Brossell summarizes:
we know that for a valid test of writing performance, multiple writing samples written on different occassions and in various rhetorical modes are preferable to single document drawn from an isolated writing instance (qtd in Yancey 491).
This method, collecting a number of pieces written in real rhetorical situations, rated by teachers of writing as either pass / fail, rather than the holistic scale of, for example, 1-6, improved the reliability of the testing by redefining the term within this context. According to Yancey, the raters "were not trained to agree, as in the holistic scoring model, but rather released to read, to negotiate among themselves, 'hammering out an agreeable compromise.' (492).
In this way reliability is increased through the negotiation of what Peter Elbow terms "community standards" (Elbow, Portfolios, xxi). Throughout the development of writing assessment, classroom teachers had lamented the fact that what was done with assessment did not relate to what they were doing in their classes creating dissonance between testing and practice. Finally, with Elbow and Belanoff and others developing the portfolio as a model for placement and program assessment, as well as classroom assessment, the two were merged: knowledge is created about assessment, but also about, and informing practice as well.
At this point, I might introduce information from SWIC's basic writing assessment program.
posted by CA Lund @ 1:40 PM
0 Comments:
Post a Comment
<< Home