Developing an EAP test for undergraduates at a national university in Malaysia: Meeting the challenges (Part 2 of 2)by Mohd. Sallehhudin Abd Aziz (Universiti Kebangsaan Malaysia ) |
In addition to correlating the sub-tests with each other, the sub-tests were also
correlated with the test total scores. The findings are illustrated in the table below.
| Subtest | Overall test scores |
| Listening | 0.79** |
| Reading | 0.53** |
| Writing | 0.54** |
| Speaking | 0.73** |
Table 5 illustrates the correlations between the sub-tests and the overall test. It is
apparent that the correlations were on the positive side. The correlations between the
sub-tests and the total test scores were fairly high because the total score was more of a
general measure of language ability rather than an individual score. The highest correlation
(0.79) was between listening sub-test and the overall scores. Another
high correlation was with the speaking sub-test (0.73) and the overall test. Although the
correlations for reading
and writing sub-tests were lower than listening and speaking, they were still fairly high.
In a study by Alderson, Wall and Clapham (1986), the correlations between the sub-tests and the total scores were
from 0.66 to 0.72. One aspect that can be deduced from this data is that there
is a strong correlation between the sub-tests and the overall test scores. This positive
correlation is a good indication the concurrent validity of the proposed test.
Another procedure for ascertaining construct validity is to correlate sub-tests with total
scores minus self (Alderson et al., 1995). This was also done in this study. Table 6 shows
the correlations. These were expected to be lower since the overall test score was a measure
of individual component score rather than general measure of language ability. It is
obvious that the highest correlation is with listening sub-test (0.49).
A 0.43 correlation was found with speaking sub-test (0.43) and the total score minus self.
This suggests that these two sub-tests have a reasonably strong effect on the overall total scores.| Subtest | Total minus self |
| Listening | 0.49** |
| Reading | 0.28* |
| Writing | 0.23* |
| Speaking | 0.43** |
[ p. 142 ]
In short, the data gathered from the statistical analyses has provided
ample evidence of the construct validity of this test. Now let us examine
this test from a qualitative perspective.
In this study, construct valididation was also made by a priori
non-statistical methods. According to Bachman
and Palmer (1996, p. 96) in determining the construct validity of a test whose purpose
is to make inferences about students' future performance, "it is essential to engage the
expertise of a subject matter specialist in the design and the development of the language
test". Taking this view in mind, six specialists were asked to judge the construct of the
test. The specialists were made known explicitly the construct of the proposed test and
asked to make judgments after an inspection of the test with regard to its construct
validity. This concerns more of a priori validation of the test.
This non-statistical approach of judgmental analysis has been utilized by researchers
such as Popham (1978, 1981), Hudson and Lynch (1984a), Davidson et al. (1985), Wong
(1992) and Alderson et al. (1995). Although this procedure of investigating construct
validity is subjective in nature, it is now widely accepted. Ebel (1979, p. 304)
argues, "one should never apologise for having to exercise judgment in validating a
test. Data never substitute for good judgment".
This pragmatic approach to construct validation was strongly adopted in the study.
The main reason why this qualitative judgment was utilized aside from the quantitative
data is that the test is basically a performance test, which is better suited with this kind of
analysis. In the study, the non-statistical aspect of validation was ensured by making sure
that the test specifications were constructed based on the candidates future language
use in a way that closely adhered to it.
How the the experts rate this test?
A great majority (83%) believed that the language ability construct of
the test was 'Very clearly defined'. Only 17% maintained that it was 'Moderately defined'.
An average of 67% believed that all the sub-tests of listening, reading,
writing and speaking clearly reflected the target language use setting.
33% of the experts also felt that the test task was 'Moderately reflective' of the
target language use situation. None of the respondents chose the 'Somewhat reflective'
or 'Not reflective' categories. Though only six experts were involved in this evaluation,
the majority did feel the test task clearly reflected the target language use situation.
The evidence gathered so far shows supports the construct validity of this test.
[ p. 143 ]
Concurrent validity
This type of validity involves collecting other external measures at the same time as the
the experimental test, or possibly at a later time. The first method used in
the study to measure concurrent validity was by correlating the scores of the incoming
students in the proposed test with other tests, namely the SPM English 1322 and the SPM English 1119
(both widely used tests in Malaysia) that the subjects had taken prior to entering the university.
The second way of determining criterion related validity was to gather the opinions
of those in regular contact with law students. In this study, the most appropriate persons
to do that were the law and language instructors. These persons were asked to rate
the students' language ability and find out whether the tests scores matched the
teacher's measures of the students' ability. In short, to determine the concurrent validity of
the instrument, a co-relational study between the students' tests scores in the proposed
instrument with that of other measures was taken concurrently.
The third method used to complement the first set of data gathered for the concurrent
validity of this study was through self-assessment. The students were asked to rate
themselves in terms of their language ability and assess their general language proficiency.
Their tests scores were later matched with their own self-assessments. This method of
determining concurrent validity was also used by Wall et al. (1991). They have shown
that there was a reasonably close relationship between the self-assessments and the
NCE scores. Criper and Davies (1988, p. 52) found a 0.39 correlation between students'
self-assessments and test scores. Wall et al. (1994) on the other hand managed to get
correlations of 0.30, 0.41 and 0.51 for writing, reading and listening respectively.
Specifically the external criteria used for establishing concurrent validity in this
study were:
The correlation figures between this test (the overall test scores and the sub-tests
scores) and the SPM English 1322 test are given in Table 7. The figures show that the correlation
between the test and the SPM English test was quite high (0.62). This is not surprising
because both tests have the same purpose — assessing students' language ability — and
they assess all the language skills although the skills are not explicitly demarcated in the
SPM English 1322 test. In addition, they are also integratively tested in the exam.| SPM 1322 English | |
| Overall test | 0.62 ** |
| Listening sub-test | 0.39 ** |
| Reading sub-test | 0.23 |
| Writing sub-test | 0.31 ** |
| Speaking sub-test | 0.66 ** |
[ p. 144 ]
In validating the Bogazici English Language Test, Hughes (1988) correlated the total
scores of that test with those of the Michigan Test of English Language Proficiency,
obtaining correlation coefficients between 0.70 to 0.84. These correlations were quite high
considering that the purpose of the two tests were similar, but methods of testing
were quite different. Even though the overall test correlated relatively high with the SPM
1332 English test, this was not the case with the sub-tests. The low correlations, however,
might be explained by the different aspects of language being tested. The correlation between
writing sub-test and SPM English 1332 test was just 0.31 – a figure which
does not differ much from Wong's (1992) 0.27 correlation between a writing test and the SPM English 1332 test.
To recap, the overall test and the
SPM 1322 English were found to be somewhat highly correlated (0.62). This statistical
evidence gathered clearly provides some support for the concurrent validity of this test.| 1119 English | |
| Overall test | .42** |
| Listening test | .08 |
| Reading test | .13 |
| Writing test | .23 |
| Speaking test | .60 ** |
As presented in the table above, the correlation between the test scores and the
SPM English 1119 test scores are significant at the 0.01 level. Nonetheless, it is a comparatively
low correlation of 0.42.
This figure is somewhat below the correlation coefficient for the SPM 1332. A likely reason
for this difference is the slightly different aspects of performance were measured.
The SPM English 1119 test is a more literature-based reading and writing test designed for
students planning to study in the United Kingdom. On the other
hand, the SPM English 1332 test is considered a more general test for a wider group of candidates
and based on the national Malaysian KBSM syllabus.
The language instructors assessed the candidates' general proficiency
and their ability in listening, reading, writing, and speaking on a five-point scale. The scale
was designed to get the opinions on the students' overall language ability as well as their
individual listening reading, writing and speaking skills. Table 9 represents the correlations
between the students' test scores and the assessments of language instructors.
[ p. 146 ]
| Language instructor's assessments of students' general proficiency | |
| Overall score | 0.75 ** |
| Listening sub-test | 0.75 ** |
| Reading sub-test | 0.43 ** |
| Writing sub-test | 0.26 |
| Speaking sub-test | 0.49 ** |
The results indicate that there was a close relationship (0.75) between the language
instructors' global assessments of the candidates' language ability and the students' total
scores on the test. The correlation between the test and the instructor's judgments
was higher than the correlations between the test and the SPM English language
results. This is not unusual considering that the language instructors were
closer to the students. Since they they knew their students
well and saw them frequently, they could judge their students' English language
proficiency with more accuracy.
The correlations between the instructors' judgments and the reading, writing and speaking sub-tests were
rather low. Table 9 indicates these correlations range from 0.26 to 0.49. Criper and Davies
(1986, p. 63) in the Edinburgh ELTS Validation Project also had low figures for language tutors'
judgment based skills and overall test (0.35 -0.43). Unexpectedly, language instructors'
judgments fit in more closely to the listening sub-test with a correlation of 0.75. However,
there may be an explanation for this. It has to be pointed out that the language instructors
could not directly assess the students' listening skill as compared to reading, writing and
speaking. As such, they might have difficulties in assessing the students' ability in that
skill area.
In summary, the correlation between the overall test scores and the language
teachers' judgments of the students' general proficiency is strong. In fact, it is higher
than the correlations with the SPM 1119 and 1332 English tests. This speaks
well for the concurrent validity of the test.
*NUMBER* specialists were asked, in the second half of the semester, about their judgments
of the students' overall language proficiency and their individual skills. The judgments
were important as they could indicate whether the test was measuring the students'
language ability to cope with the academic requirements of the course. Table 10 shows
the correlations between subject specialists' judgments and the students' overall
scores on the test.
[ p. 145 ]
| Content specialists' assessments of students' general proficiency | |
| The test overall | 0.61** |
| Listening sub-test | 0.52 ** |
| Reading sub-test | 0.21 |
| Writing sub-test | 0.19 |
| Speaking sub-test | 0.60** |
As expected, the figures show a reasonably high correlation (0.61) between the
test as a whole and the global assessments of the students' general proficiency by
the subject specialists. Even though the correlations were somewhat high, they were consistently
lower compared to the assessments given by the language instructors. Weir (1988, p. 68)
in validating TEEP also reported similar findings, "it does seem likely that the English
language teachers cooperating in the concurrent validity study were . . . better able to
assess a student's language proficiency than subject tutors".
Table 11 shows the correlations between the students' overall and sub-test scores and
their assessments of their own general English proficiency.| Student self-assessments of general proficiency | |
| The test overall | 0.68** |
| Listening | 0.64** |
| Reading | 0.37** |
| Writing | 0.23 |
| Speaking | 0.48** |
It is apparent that the correlation between the overall test scores and the students'
own assessments of their own general proficiency is higher than the sub-tests. The correlation
between the test's overall scores and the students' assessments of their general
proficiency was 0.68 and was not unexpected. However, unexceptional levels were
found in the relationship between the sub-tests and the students' assessments of the
global proficiency. It can be seen from
the table that the correlations were between 0.23 and 0.48. Nevertheless, the greatest
congruence was in the area of listening sub-test (0.64). One factor
influencing the magnitude of the correlation coefficient was the inability of the students
to evaluate their listening skills accurately.
[ p. 146 ]
Based on the quantitative and the qualitative data gathered, there
was strong evidence of the concurrent validity of this test.
This involves gathering of subsequent external measures that could indicate how well
the test predicts the intended performance. According to Hughes (1989, p. 25), predictive
validity "concerns the degree to which a test can predict candidates' future performance".
It is normally a posteriori investigation. Traditionally, testing developers have asked language students,
teachers, and subject specialists to make global assessments of the students' abilities
(Davies, 1965; Moller, 1982; Wong, 1992).
The external measures used in the study were:
Table 12 summarizes the correlations between the test overall and sub-test scores and
the English for Law results. This had the highest correlation (0.88) compared
to all other external criteria discussed earlier in determining concurrent validity.| English for Law Test scores | |
| Overall test scores | 0.88** |
| Listening sub-test | 0.73** |
| Reading sub-test | 0.39** |
| Writing sub-test | 0.53** |
| Speaking sub-test | 0.62** |
The strong figures between the two tests indicate that they are measuring the same construct.
Both tests attempt to measure students' language ability.
[ p. 148 ]
| R | R2 | Adjusted R2 |
SEE | |
| Overall test | .885 | .784 | .781 | 8.2011 |
| Listening sub-test | .733 | .537 | .532 | 11.9917 |
| Reading sub-test | .397 | .157 | .147 | 16.186 |
| Writing sub-test | .868 | .753 | .747 | 8.8084 |
| Speaking sub-test | .894 | .800 | .793 | 7.9835 |
The data from regression analysis suggest that the listening writing and speaking
sub-tests are good predictors of the students' ability.| Language instructors assessments of students' academic potential | |
| Overall test scores | 0.80** |
| Listening sub-test | 0.76** |
| Reading sub-test | 0.47** |
| Writing sub-test | 0.39** |
| Speaking sub-test | 0.48** |
As indicated above, the correlations between the overall test scores and the
language instructors' assessments of students' academic potential was
0.80. The strong correlations between these two measures provided some evidence of
the reasonable predictive validity of the test. As expected, the sub-tests generally did not
correlate closely with the language instructors' estimates.
As shown in Table 15, the correlation between the test and the subject specialists'
estimates is also significant at the 0.01 level. Nonetheless, it must be pointed out that the
correlation was comparatively moderate at 0.44. This is a far contrast from the correlation
achieved from the language instructors' estimates. One possible explanation for the low
correlation could be due to the lenient and inaccurate assessments given by the subject
specialists. The subject specialists were perhaps too conservative and did not want to
rank the students very low or fail them.
[ p. 149 ]
| Subject specialists' assessments of students' academic performance | |
| Overall test scores | 0.44** |
| Listening sub-test | 0.39** |
| Reading sub-test | 0.29** |
| Writing sub-test | 0.06 |
| Speaking sub-test | 0.37 |
As presented in Table 16, the correlation between the overall test scores and students'
own assessment is higher than the ones given by the subject specialists — 0.51 compared
to 0.44. One interesting finding is that the test scores were more closely related to the
language instructors' estimates of academic potential than the subject specialists and
the students' themselves.| Student self-assessments of academic potential | |
| Overall test scores | 0.51** |
| Listening sub-test | 0.43** |
| Reading sub-test | 0.22* |
| Writing sub-test | 0.24* |
| Speaking sub-test | 0.39* |
One would have expected that the correlation would be higher for the subject
specialists' estimates since they could confidently estimate whether a student could
pass or fail a course since they were aware of the standards and the expectations of a
particular course they were teaching in the university. As mentioned earlier, one reason for
the moderate correlation was seemingly lenient assessments given by the specialists.
As expected, the results of the correlations between the sub-tests and the students'
estimates of their academic potential did not show any surprises. The correlations were
all generally low. The lowest correlation was with the reading sub-test (0.22).
All in all, the figures established that the test correlates moderately with the estimates
given by the language specialists, the subject specialists and the students. There is a
strong correlation (0.80) between the overall test scores and the language instructors'
assessments of the students' academic potential. The correlation against subject
specialists' estimates is also significant albeit slightly lower. The data gathered have
indicated strong evidence of the predictive validity of the test.
[ p. 150 ]
Reliability
Reliability is another important characteristic of a good test. Weir (1990, p. 32)
suggests that there are basically three procedures that can be used to obtain measures
to estimate reliability. They are i) test re-test reliability, ii) inter-rater reliability and iii)
internal consistency reliability measures. With regard to reliability, this study focused only
on inter-marker reliability. Inter-rater reliability was established by correlating the scores
obtained by candidates from the markers. The reliability for the raters was calculated
using the Pearson product moment correlation formula. The Table 17 illustrates the
validity/reliability and procedures used to validate the proposed test.
Table 17 shows very strong inter-rater reliability coefficients. All the raters were almost
equally consistent with each other.| Rater A | Rater B | Rater C | Rater D | |
| Rater A | – | 0.936** | 0.966** | 0.953** |
| Rater B | 0.936** | – | 0.934** | 0.879** |
| Rater C | 0.966** | 0.934** | – | 0.953** |
| Rater D | 0.953** | 0.879** | 0.953** | – |
In short, the inter-rater correlations for this test was very
high. The high correlations coefficients are good indictors of the reliability of the test.| "This study is probably one of the first attempts at designing, constructing and validating an English language test for incoming law students." |
This study is probably one of the first attempts at designing, constructing and validating
an English language test for incoming law students. Even though it has some
limitations, the study nonetheless has provided strong and solid empirical evidence
about the performance of the test. In short, the test has shown that it is a good English
language test for academic purposes. The satisfactory reliability and validity evidence
gathered have proven that this is a reasonably good test and that it can clearly stand on
its own. This test clearly could be used as a model test or as an instrument for deciding
whether incoming law undergraduates have the necessary language skills to pursue studies in law at UKM.[ p. 151 ]
Return to Main Article (Part 1)
Appendices
Topic Index
Author Index
Page Index
Title Index
Main Index
Topic Index
Author Index
Page Index
Title Index
Main Index[ p. 152 ]
