Communicative assessment in a personal curriculum at Kanda University of International Studies

Curriculum Innovation, Testing and Evaluation: Proceedings of the 1st Annual JALT Pan-SIG Conference.
May 11-12, 2002. Kyoto, Japan: Kyoto Institute of Technology.

Communicative assessment in a personal curriculum at
the Kanda University of International Studies

by Alistair Van Moere & Francis C. Johnson (Kanda University of International Studies)

The colloquiums at the 2002 JALT Pan-SIG Conference revealed that institutions throughout Japan are seeking ways to select or design tests to accurately stream and place students, and assess their progress, not in terms of just achieving course-credits but rather in increasing language competence. This paper provides an overview of a proficiency test and how it relates to the curriculum at Kanda University of International Studies (KUIS) in Chiba.

". . . the university is working towards creating a highly individualised curriculum in which each student will be required to study in proficiency courses only until they reach a demonstrable level of competence."

KUIS streams incoming students and measures their progress annually using its own video-mediated proficiency test, which includes essay and oral components. In a major curriculum restructuring, the university is working towards creating a highly individualised curriculum in which each student will be required to study in proficiency courses only until they reach a demonstrable level of competence. Using these test scores as well as other measures of competency, students who attain the required standards will be able to effectively bypass much of the 1st and 2nd year Basic English Curriculum, and instead earn their requisite credits at the Advanced (3rd and 4th year) levels, where courses are more content-oriented.

Streaming and assessing proficiency increases with KEPT

KUIS is a private university with an annual intake of around 450 English majors. These have traditionally been grouped into mixed-ability classes of 25-30 students, but in line with curriculum revision it was decided that the 2002 intake would be streamed in descending order of proficiency. The university's own fully operational, curriculum-linked test – in development since 1989 – seemed the perfect vehicle for placing incoming freshmen. Since it's inception the Kanda English Proficiency Test (KEPT) has been used as a research tool and measure of students' English proficiency increases throughout their university studies (Gruba,1993; Bonk, 2000). Unlike the TOEFL^® or TOEIC^®, the tasks and texts featured in the KEPT are closely related to the university's English curriculum. It therefore mirrors the tasks on which KUIS expects students to be able to perform well. The size and scope of the KEPT deserves some explanation.
Each year some 1,600+ students from various departments take this test, making it one of the largest institutional EFL tests in Asia. Each test cycle consists of two administrations, firstly as part of year-end finals for 1st and 2nd years, and secondly two months later in March for incoming freshmen.
KEPT currently exists in four complete forms (with a fifth being written), meaning that students may sit a different form the test on arrival at Kanda University, and at each year-end of their studies. They are administered in five-year cycles so the 2002 test, for example, will be re-used in 2007. The advantages of having different forms of the same test are (i) one test can be safely used twice without examinees cheating, and (ii) the cycle of post-test item-analysis and revisions pushes up test reliability each year.

[ p. 155 ]

The KEPT has several outstanding features. It is video-mediated, with video-voice instructions and an on-screen timer display performing the role of proctor. The video- listening passages consist of a number of short skits, with actors who negotiate obstacles while traveling in another country. Furthermore, the test has five subsections which are connected thematically by issues that students can relate to, such as travel, homestay, dating, marriage, drinking, and friendship.

The norm-referenced Reading, Grammar and Video-Listening subsections each contain several graded passages and 25 multiple-choice items.
The Writing section allows students 30 minutes to write an academic essay in response to a prompt (for rating criteria see Appendix 1).
The Oral section is in the form of a group conversation. Students are arranged into groups of four and given a prompt to discuss for 10 minutes (for rating criteria see Appendix 2).

While the first three sections are computer-read, all essays and oral tests are rated by two teachers.
KUIS has a complement of thirty full-time lecturers working in its English Language Institute making this huge undertaking possible. In January 2002 this department proctored and rated 1,100 examinees in just three days, plus one half-day for oral/essay norming sessions. On each of two days 550 students sat the paper test in the morning, followed by group oral sessions throughout the afternoon (13 pairs of raters, with each pair assessing a group of four students every 15 minutes, can accomplish the task in under 4 hours). The third full day was left for essay rating (about 80 essays per rater, allowing all essays to be double-rated). Previously, either RASCH models were applied to take rater leniency-harshness into consideration; or the two ratings were simply averaged. As most recent analyses show that rater discrepancies are increasing, all future ratings are to be RASCH-moderated.
Scores are usually posted online on the KEPT website within one week of administering the test. Privacy is ensured by having each student log onto the site using their name and university ID number, from where they may view their own scores on all subsections and, for comparison purposes, the mean scores that were achieved by their peers (see Appendix 3).
Tables 1-3 show results for outgoing first and second year students (January 2002) and for incoming first year students (March 2002). Reported scores are standardised. The multiple-choice sections of all different forms of this test are brought into line using equipercentile equating¹, encouraging students to chart their year-on-year progress by directly comparing their scores on different tests. In Appendix 3, for example, Kayako can safely assume that her Grammar proficiency has increased over the year 2001-2002, but her Reading has not, and because of equi-percentile equating she need not worry that there was a shift in difficulty level between the Grammar or Reading subsections of the two different test versions.
The next section describes in more detail the context in which KEPT is used, and how it will be used once proposed curricular revision for English proficiency takes place.

[ p. 156 ]

Table 1: 2002 Score results of incoming first year students on all sub-tests. (n = 586)

	Reading	Grammar	Listening	All Multi-choice Sections	Writing	Speaking	Total
# of items	25	25	25	75	20	20	100
Mean	10.9	11.4	11.4	33.7	8.3	10.2	45.3
Median	10	11	12	33	8	10	45
Mode	9	8	13	30	8	10	46
SD	3.82	3.87	4.06	9.83	3.01	2.81	11.05
Kurtosis	-0.210	-0.225	-0.420	0.458	0.665	0.886	0.826
Skewness	0.480	0.395	0.194	0.363	0.592	0.110	0.611
Range	19	18	19	57	17	19	83
Low/High	4 - 23	4 - 22	4 - 23	14 - 71	1 - 18	1- 20	17 - 90
Reliability	.71 *	.68 *	.62 *	.84 *	.51**	.70**	(add)

Table 2: Score results of outgoing 2002 first year students on all sub-tests. (n = 602)

	Reading	Grammar	Listening	All Multi-choice Sections	Writing	Speaking	Total
# of items	25	25	25	75	20	20	100
Mean	12.9	12.9	13.3	39.1	11.3	12.9	55.4
Median	14	12	14	39	12	11	55
Mode	15	12	15	36	12	12	59
SD	3.7	3.8	3.9	9.11	2.8	2.0	9.6
Kurtosis	-0.522	-0.193	-0.356	-0.001	0.556	-0.297	0.245
Skewness	-0.223	0.119	-0.086	0.012	0.247	0.008	0.068
Range	18	22	23	55	17	13	61
Low/High	3 - 21	3 - 25	3 - 25	14 - 69	3 - 20	7 - 20	26 - 87
Reliability	.68 *	.67 *	.64 *	.83 *	.49**	.72**	add

Table 3: 2002 score results of outgoing 2nd year students on all subtests. (n = 446)

	Reading	Grammar	Listening	All Multi-choice Sections	Writing	Speaking	Total
# of items	25	25	25	75	20	20	100
Mean	14.6	14.5	14.5	43.7	12.1	13.4	60.7
Median	15	15	15	43	13	12	61
Mode	15	15	15	42	13	12	62
SD	3.88	3.45	3.97	8.95	2.7	2.1	9.4
Kurtosis	-0.174	-0.133	-0.039	0.009	1.077	-0.006	-0.220
Skewness	-0.111	-0.161	-0.210	-0.167	-0.061	-0.001	-0.137
Range	20	18	22	55	15	15	54
Low/High	4 - 24	4 - 22	3 - 25	16 - 71	4 - 19	5 - 20	34 - 88
Reliability	.71 *	.64 *	.66 *	.83 *	.46**	.75**	add

* Based on KR-20.
** Grand mean of Simple Agreement Coefficients for the five subscales of this test.

Towards a personal English proficiency curriculum

The present KUIS curriculum has a set number of courses which students take in the same prescribed order over a period of four years. The proposed changes affect both the course structure system and the way students progress through that system. It provides a personal curriculum for each student at KUIS. These changes enable the school to offer two types of English proficiency courses – basic and advanced.
Basic English Proficiency courses aim to develop students' ability to use English to a point where they reach a prescribed level of achievement. Students will continue to take the basic courses until they reach that level. Two types of courses are proposed – basic integrated skills courses and basic individual-skills courses.

[ p. 157 ]

When students have reached the prescribed level of proficiency they will be permitted to take advanced proficiency courses. These courses aim to maintain and improve an attained proficiency. Two types of advanced proficiency courses are being developed-content-focussed integrated skills courses and specialized skills-based courses.
Because students enter the university with vastly different levels of English proficiency² it can be assumed that they will reach the specified level at different times. As described in Part II, the system has been changed to facilitate this. KEPT is used to measure entering proficiency and students are streamed into Basic English Proficiency Classes.

Movement from Basic to Advanced courses

A key concept in the proposed changes is the measurement of the prescribed level of proficiency required for a student to be able to move from basic to advanced course. Behind this concept is the notion of the desirability of being able to certify that every graduate of KUIS will have attained a specified level of proficiency. The proposed change sets up two criteria to measure attained proficiency – a measure of global English proficiency, and a set of demonstrated English competencies.
A measure of global English proficiency is the ability to use English communicatively in terms of internationally-accepted standards. To warrant that a student has achieved a score of 800 on the TOEIC^® or 550 on the TOEFL^® is to make a statement in terms of their global proficiency. All measures of global proficiency have limited usefulness. For example, a specified score on TOEFL^® may inform that a student can be admitted to a US college and will not be required to take preliminary classes in English proficiency. The score says nothing more than that. It does not give any indication of how proficient the student is in other skilled uses of English. However, a TOEFL^® score is a publicly- recognized criterion of English proficiency. It is desirable for KUIS graduates to have such recognition, and for the university to be able to declare publicly that all its graduates will have achieved the specified standard.
One of the tasks of the KEPT Research Project is to benchmark the KEPT with such well-known tests as the TOEIC^® and TOEFL^®. Investigation into benchmarking these tests currently shows that KEPT can be used to accurately predict a TOEFL^® score to within between 6 and 30 points (Bonk, 2001). Movement from basic to advanced courses will therefore require a student to achieve a certain KEPT score. Research continues into what a suitable cut-off score would be, but it is likely to be significantly higher than the current standard (Van Moere, 2002).
Language competencies are the can do's of English proficiency learning and comprise a checklist of performances involving the use of English for different purposes. But, language competencies are not only indicators of levels of skill achievement. They can also guide curriculum design and instructional materials preparation through the setting of course goals and objectives. Thus, unlike test scores, they are intimately associated with classroom activities and may take the form of a portfolio of completed work.
While norm-referenced test scores say nothing about specific skills in using language, language competencies are empirical evidence of such skills. We maintain that a specified KEPT score and demonstrated competencies provide the necessary and sufficient basis of achievement of English proficiency to warrant movement from basic to advanced courses.

". . . language competencies are not only indicators of levels of skill achievement. They can also guide curriculum design and instructional materials preparation through the setting of course goals and objectives."

Different routes and rates of progress through the system

The proposal incorporates not only changes in the structure of the English proficiency course system. It also provides for changes in how the school system operates. The new system will provide opportunities for students to take, according to their differing needs, interests and preferences differing routes as they progress. This system permits every student at KUIS to have a personal curriculum. At the heart of a personal curriculum is the concept of learner choice. The system described above features choice in both the basic and advanced courses.

[ p. 158 ]

Learner choice will be manifest in several ways. Firstly, even in the compulsory Basic English Proficiency Course every unit in the course will be designed to provide for students a choice of classroom activities. Secondly, students will not only be able to choose different courses to take, but be able to select different types of course delivery for the same course. For example, in the current development of the course Basic Reading Skills, two modes of delivery are being used – the classroom mode and the Internet mode. It is planned to offer this course in the 2003-2004 academic year in three delivery modes – classroom, Internet, and self-access.
Thirdly, courses, and instructional units within courses, will be designed to accommodate the fact that students will, if allowed, progress at different rates. This helps solve the dilemma faced by all teachers using class- or course-based curricula, which is when to move the class on to the next unit in the curriculum knowing that some students are not 'ready' to progress while others have been 'ready' for some time.
In summary, while the proposed revisions for a personal curriculum have yet to come into effect fully, many of the provisions have already been met in terms of course delivery and materials. The extra requirement of meeting a challenging KEPT score before progressing from Basic to Advanced courses will furthermore ensure all KUIS graduates have attained a prescribed level of English proficiency.

Notes

For an explanation see http://www.arts.auckland.ac.nz/edu/staff/jhattie/aaaaa%20to%20be%20removed/Equating.html[Expired link]
In the 2002 administration of the KEPT for entering freshmen scores ranged from 17 to 90 (see Table 1).

The KEPT website is at http://kandaeli.brinkster.net/

References

Bonk, W. (2000). KEPT Argentina-2000 administration: report and analysis. Kanda University of International Studies Research Institute of Language Studies and Language Education Working Papers, 11 163-224.

Bonk, W. (2001). Predicting paper-and-pencil TOEFL scores from KEPT data. Kanda University of International Studies Research Institute of Language Studies and Language Education Working Papers, 12 65-86.

Gruba, P. (1993). A video-mediated communicative test of proficiency. Kanda University of International Studies Research Institute of Language Studies and Language Education Working Papers, 4 101-120.

Van Moere, A. (2002, Winter). KEPT score exit criteria: Interim report. Kanda University of International Studies Research Institute of Language Studies and Language Education Working Papers, Vol?? Number?? pages??.

Appendix 1 Appendix 2 Appendix 3

2002 Pan SIG-Proceedings: Topic Index

Complete Pan SIG-Proceedings: Topic Index

[ p. 159 ]