May 22-23, 2004. Tokyo, Japan: Tokyo Keizai University.
Triangulating perceptions of learners' needs:
|
|
Abstract Keywords: needs analysis, data triangulation, program evaluation, survey design
|
Triangulating perceptions of learners' needs:
|
|
Abstract Keywords: needs analysis, data triangulation, program evaluation, survey design
|
How can teachers and program administrators evaluate the extent to which a given curriculum is meeting students needs? This paper explores the interface between curriculum design and perceptions of students needs by consulting three different populations at one university EFL program in Tokyo.
| ". . . needs analysis should be a logical first step in curriculum development." |
Needs analysis refers to "the activities involved in gathering information that will serve as the basis for developing a curriculum that will meet the learning needs of a particular group of students" (Brown, 1995; p.35). Usually, it involves examining both qualitative and quantitative information based on questionnaires, tests, interviews and observations.
It is argued that needs analysis should be a logical first step in curriculum development. For example, in Brown (1995)'s systematic curriculum development model (Figure 1), needs analysis is perceived as the first component, followed by five other components, "goals and objectives", "language testing", "material development", "language teaching", and "program evaluation".
[ p. 33 ]
Discussing all six components mentioned in Fig. 1 is outside the scope of this paper. According to Brown (1995, p.21), a logical outcome of the thorough study of learners' needs in a language program should be twofold. First, goals should be specified so that general statements can be made about what is to be accomplished. Second, precise statements about what content or skills students must master to attain those particular goals should be made.
With a thorough needs analysis, all other activities in the curriculum development process may more readily be accomplished. Needs analysis also helps all the other parties (i.e., test developers, material developers, actual teachers, etc) focus on appropriate content. Such data, for example, would inform instructors on how to go about their practice in the classroom.
Finally, how does one carry out an effective needs analysis, which can be the basis of a sound curriculum development? In the following section, some methodological issues in needs analysis are discussed.
Issues regarding multiple methods and sources
It is often the case that classroom teachers conduct small-scale studies using informal questionnaires. This may offer some information about the sample population, but is subject to factors which skew the data. For example, respondents often write what they perceive a surveyor wants them to write. And if a survey is long, then they will tend to rush through it - not paying much attention to the questions near the end.
| ". . . needs analysis research should be carefully designed to include multiple data sources and methods." |
Information from such informal needs analysis needs to be questioned critically. Researchers such as West (1994) Brown, (1995) Long (1999), Witkin and Asltschuld (1995) argue that needs analysis should use multiple methods and multiple sources to increase its overall reliability and validity. For example, Witkin and Asltschuld (1995, p.279) state:
We do recommend that you use more than one data source or method and that you balance quantitative methods with qualitative ones. Data from any single method (surveys, interviews, focus groups, or analysis of existing records) are generally insufficient to provide an adequate basis for understanding needs and making decisions on priorities
This paper argues that needs analysis research should be carefully designed to include multiple data sources and methods. To collect information from various kinds of participants, one would do well to compare data from several sources.
The next section will briefly mention one study using multiple data sources conducted at a 4-year private university in Japan. In this study, I used semi-structured interviews, open-ended and closed-response questionnaires with teachers, students, and program coordinators and classroom observations in this particular university's language program. The hope is that readers will conceptualize ways to go about conducting similar studies for the programs at their universities.
An example using this approach
In 1992 the department of English at Aoyama Gakuin University in Tokyo established an English program known as Integrated English (IE) for first and second year students. Students in this program are divided into three levels based on their scores on their in-house English proficiency test and take IE core (180 min), IE writing (90 min), and IE listening (90 min) courses weekly.
English is used as the medium of instruction in the IE core course and students are expected to speak English at all times. In the IE writing course, students are taught paragraph and essay writing. In the IE listening course, students watch a variety of videos, usually documentary films or excerpts from movies and TV dramas. In class, they are asked to complete work sheets with listening comprehension questions. Japanese instructors teach all IE listening sections. Except for the classes that non-Japanese instructors teach, Japanese is usually the medium of instruction in the IE listening classes. Evaluation reports done in May 1999 suggested there was a strong need to revise the IE program, especially the listening section (Strong, 1999). Therefore, further research was needed to gather information about how to revise that program. In order to better understand learners' needs, I conducted a needs analysis study for this program.
[ p. 34 ]
In this study, the following methods were used: semi-structured interviews, open-ended and closed-response questionnaires (see Questionnaire A and B), as well as observations. Three types of respondents were used in this study: students, teachers, and program coordinators. Based on an intensive review of the literature concerning needs analysis, I devised the question types shown in Table 1. I attempted to obtain the data using semi-structured interviews, open-ended and closed-response questionnaires (Questionnaire A and B), and observations from the three populations mentioned above to obtain a clearer image of learners' needs.1. TARGET TASKS: What kinds of things would you like to do in the future using English? 2. PROBLEMS: What kinds of things would you have difficulty with in listening? 3. PRIORITIES: What kinds of things would you prefer to listen to in the IE listening class? 4. ABILITIES: What kinds of things can you do using English now? 5. ATTITUDES: What do you like about and do not like about your IE listening class? 6. PROBLEMS: Do you have any complaints of problems in the IE listening class? 7. SOLUTIONS: Do you feel that your IE listening class helps you to be a better listener? If not, do you have any ideas about certain things that need to be changed in IE listening program?
In the Spring 2000 IE listening program, there were 663 students enrolled and 18 instructors. Among the 663 students, 585 (80% female) students participated in this study and responded to either Questionnaire A (222 students) or B (370 students). Of these 585 students, 18 were randomly chosen to be interviewed, 15 of which actually participated, 80% of which were female. Those students whom I interviewed were randomly chosen from the class lists and also responded to Questionnaire A or B. The instructors who taught IE listening were all native speakers of Japanese, except for one instructor, who was British. Two program coordinators, who were native speakers of English, originally from Canada, also participated in the interviews.
At the beginning of this study, I observed nine IE listening classes and submitted Questionnaire A to the students in those classrooms. I observed each class with the instructors' permission and took field notes to record what was happening in the class. At the same time, Questionnaire A was administered either at the beginning or the end of each class. When I administered Questionnaire A, I simply went into six classrooms, explained the purpose of my research, and gave questionnaires to the students.
In an attempt to avoid some questions being left unanswered, I read the questions aloud, clarified possible misunderstandings and went around the class to make sure the participants were filling in the form. Questionnaire A took students about 10 minutes to fill out. All 222 students who attended class on that day responded to Questionnaire A.
Next, I conducted interviews with some of the subjects. I planned to conduct semi-structured interviews for 10-15 minutes each with eighteen students, six teachers, and two program coordinators. A portable recorder was used to record the interviews. Six question types were designed to determine learners' needs in the following areas: (a) target tasks, (b) problems, (c) priorities, (d) abilities, (e) attitudes, and (f) solutions. I conducted the interviews for 10-15 minutes with (a) the fifteen students, who were selected randomly from each class (b) six IE listening teachers, and (c) two program coordinators. I was unable to conduct the interviews with three of the students because they did not show up for their interview appointments. For those teachers who were unable to participate in the interviews, I asked them to fill out Questionnaire A. I placed the questionnaires and return envelopes in their mailboxes. Nine of twelve teachers returned their questionnaires, a return rate of 75%.
Then, all students who were not given Questionnaire A were given Questionnaire B at the beginning of July. After revising the questionnaire format as suggested by a program coordinator and professional colleagues, I visited twelve classrooms and requested the teachers' cooperation. Although one teacher declined to administer the questionnaires, eleven teachers agreed to do so. In four of the classrooms I personally administered the questionnaires, while seven teachers conducted the questionnaires themselves in the remaining classes. Among the 417 students who were enrolled in these twelve classes, 370 students answered the questionnaires (return rate= 89%). Three questionnaires were discarded because they were not filled out. Although some questionnaires were incomplete data from the answered questions was included in this study.
The results of this research, which includes both qualitative and quantitative analyses as shown in Tables 2 and 3, are presented below.
[ p. 35 ]
In Table 2, the means and standard deviations (SD) of the students' answers are presented. A higher mean shows that more students agreed that they would use English in that particular activity. Likewise, a lower mean suggests less intention to use English for that particular purpose. The percentages selected for each response (1, 2, 3, 4, 5) are also presented to give a more precise idea of how the responses were distributed. The table is organized in mean order. Hence, responses with the highest means are seen at the top and those with the lowest at the bottom. In Table 2, I presented part of the matrix for learners' target tasks following Brown's (2001, p. 216) recommendations.
In Table 2, ten choices are shown in mean order. Three choices have means higher than 4.00: traveling [mean=4.71, SD=0.69], watching movies for fun [mean=4.22, SD=1.06], and studying abroad [mean=4.12, SD=1.02] are strongly favored choices. More than 75% of the students who responded to this question either agreed or strongly agreed with those three choices. On the other hand, becoming interpretators [mean=3.00, SD=1.30], and teaching school [mean=2.47, SD=1.35] had means lower than 3.00. Except for the choice of talking with friends casually [mean= 3.33, SD= 1.39], the choices regarding English at the workplace were less favored than the choices regarding English in personal settings.
In Table 3, representative answers obtained from each qualitative method are presented. Participants from each source shared similar points of views on learners' target tasks in the future. However, it should be noted that teacher and student views about what learners' target tasks seemed to conflict. Although students were all very eager to share their ideas about their dream job, their opinions differed from teachers concerning what was a realistic future job.
For example, one teacher commented:
In my way of thinking, there is a gap between students' wishes and reality. Most of the students want to work for international divisions in American companies, and negotiate with foreigners. That's all that they can imagine. In my opinion, as long as they work for companies in Japan, it will not matter if they are good at English or not. A small number of people can use English as teachers. That's another possibility: there are a very small number of people who apply their English skills at work. So the only chances they have to use English are if they become teachers or flight attendants. However, in my days, and these days as well, there are fewer and fewer chances to be flight attendants. You know, few people use English at work. I think that only returnees who have lived abroad for five or ten years are going to get the work that requires English. When I look around, I am the only one who uses English at work.
In contrast to most students who merely wrote about their own preferences, this teacher cited above who had graduated from this university's graduate program shared her view on students' future use of English. Because she knew others who had already graduated from the university recently, her view is worth noting. She said that only returnees who have lived abroad for five or ten years would have the opportunity to use English at their workplace everyday. This is the reality in Japan where, with the exception of infrequent communication with foreigners, conversational English outside of the classroom is a rarity.
[ p. 36 ]
On the other hand, one program coordinator mentioned:. . . you never know what's going to happen in the future. Things may change. You may all of a sudden be in a position where you need English. So just because you don't need it immediately doesn't mean you won't need it in the future.In other words, situations may arise in which English becomes suddenly necessary.
Conclusion
| ". . . it is important to involve all participants in the educational process and develop a program based on broad consensus rather than top-down management ultimatums." |
As illustrated in the last section, one can see how using a multiple methods and source needs analysis can provide researches with complex information about the needs of students. In the example above, we saw that learners, teachers, and program coordinators each have differing notions on learners' target tasks. These differing conceptions of learner's needs present the program with a daunting task. However, in this case, program coordinators and I agreed that it is also important to design a program that addresses some common desires that students expressed throughout the questionnaires and interviews. Due to the findings brought forth from this study, this program is trying to better meet these student expectations. On the other hand, it is found that some teachers may resist changing their practices, thereby making the implementation of such program-wide changes difficult.
While needs analysis studies such as this one using triangulation of sources and methods take time, resources,
and a great amount of researcher energy, they often do not result in any real changes in substantive curricula design (Altschuld and Witkin,
2000). It is hoped that the results of this study and future research in this area will improve language curriculum design in order to
more closely meet learners' needs. I am looking forward to seeing more research on how needs analysis can help bring real changes to language curriculum.
References
Topic Index
Author Index
Page Index
Title Index
Main Index
Topic Index
Author Index
Page Index
Title Index
Main Index [
p. 37
]
