The Interface Between Interlanguage, Pragmatics and Assessment: Proceedings of the 3rd Annual JALT Pan-SIG Conference.
May 22-23, 2004. Tokyo, Japan: Tokyo Keizai University.

The interface between interlanguage pragmatics
and assessment

中間言語語用論と評価の境界領域
by Andrew D. Cohen (University of Minnesota)

This article offers in written form most of the points that I made in my plenary at the JALT Pan-SIG Conference in the Spring of 2004. The article will first speak to the importance of pragmatics, and especially speech acts, in the teaching and learning of a second language (L2). Second, it will speak to the challenges of testing pragmatic behavior. Finally, suggestions for action are offered.
本稿は著者が2004年の春にJALT Pan-SIG大会の基調講演にて口頭発表した際の要 点をまとめたものである。本稿ではまず、第二言語の教育と学習における語用論、特 に発話行為の重要性について述べる。次に語用論的行動を評価する際の難点について 述べる。最後に、こうした問題に対する方策への提言を論じる。


There is a basic premise in interlanguage pragmatics — that it is not enough just to know the equivalent words and phrases in a second language (L2). Learners need to determine the situationally-appropriate utterances, namely: Yet a powerful influence is working against the appropriate application of the L2 forms — namely, how we do it in the native language and in other languages we know. The extent to which a person is able to cope with pragmatic challenges has been referred to as their pragmatic ability, namely:
the ability to deal with "meaning as communicated by a speaker (or writer) and interpreted by a listener (or reader) ...[and to interpret] people's intended meanings, their assumptions, their purposes or goals, and the kinds of actions (for example, requests) that they are performing when they speak. (Yule, 1996, p. 3-4)
Continuing with basic definitions, a speech act is an utterance which serves as a functional unit in communication. Utterances have a literal or propositional meaning (e.g., "Where was I when that cell phone rudely interrupted me?" as uttered by a speaker who was just distracted away from his talk). Utterances also have a functional or illocutionary meaning (i.e., the effect that the utterance or written text has on the reader or listener, in the cell phone instance serving as a complaint with the remedy that the participant turn it off so there will not be another similar interruption).
Another useful distinction is that of a speech act set, which has been defined as the set of functional strategies typically used by native speakers of the target language to perform a given speech act (Olshtain and Cohen, 1983). Any of the speech act-specific strategies in a speech act set might be recognized as constituting the speech act in question, depending on the situation and the cultural group involved. So, the indirect request to shut off the phone — "That cell phone ring just distracted me" — could function on its own as a complaint or coupled with other strategies within the same speech act set:

[ p. 1 ]

Pragmatic behavior in SLA

"While there has been a presumption (or perhaps wish) among language educators that learners will eventually acquire on their own a sense of how to perform speech acts just from being exposed to the target language, experience and research as well has proven this view to be false."
Pragmatic behavior, especially as reflected in speech acts, is clearly at the intersection of language and culture since it is often not enough to string a series of words together grammatically; they must be in a meaningful sociocultural context as well. For example, learners of Japanese need to know that when the proprietor of a restaurant in Japan thanks them as they are leaving the restaurant for having eaten there, it is inappropriate to say, do iteshimashita [you're welcome], which is the way that they would do it in most English-speaking cultures.
Fortunately, descriptions of speech acts keep improving, as more empirical work continues to appear. Despite some reservations about the broad terms used to describe speech act distinctions (see Beebe and Waring, 2001), speech acts have been described in terms of their sociopragmatic and pragmalinguistic nature (Thomas, 1983). In other words, researchers have collected empirical evidence as to the appropriateness of performing speech acts in certain contexts — the sociopragmatic aspect (e.g., whether/when you could ask a colleague how much he makes a month), and also just what language forms are appropriate for performing these speech acts — the pragmalinguistic aspect. The classic case of subtle pragmalinguistic distinctions is that of how to intensify an apology in American English, where "very" is more an indication of etiquette ("I'm very sorry to interrupt your meeting, but...") while "really" implies regret ("I'm really sorry I bumped into you.").
While there has been a presumption (or perhaps wish) among language educators that learners will eventually acquire on their own a sense of how to perform speech acts just from being exposed to the target language, experience and research as well has proven this view to be false. Even learners in an L2, rather than a foreign language environment, may go many years without mastering speech acts — because these utterances represent such complex speech behavior (Blum-Kulka and Olshtain, 1986). For this reason, there is a growing consensus that for best results, we may in fact need to explicitly teach L2 pragmatics (see Rose and Kasper, 2001; Bardovi-Harlig and Mahan-Taylor, 2003).

A Website on speech acts for teachers

At the Center for Advanced Language Acquisition (CARLA), Noriko Ishihara and I have developed over the last three years this source of information for language teachers, materials developers, learners, and researchers: http://www.carla.umn.edu/speechacts/teaching.html [inactive link].
This website was created in response to a felt need expressed by a curriculum writer that basic information about speech acts was not readily available, and that he did not have time to seek it out in research reports.
". . . there is a real problem associated with the explicit teaching of speech acts: the very act of doing so means that we are extracting them from their rich context, creating a more artificial situation."

The creation of the website was undoubtedly motivated in part by a desire to encourage the explicit teaching of speech acts. Yet there is a real problem associated with the explicit teaching of speech acts: the very act of doing so means that we are extracting them from their rich context, creating a more artificial situation. In addition, courses of instruction have a finite time period, with much to cover and teachers are usually untrained with regard to speech acts. Thus, teachers would be hard pressed to cover even the basics of the host of relatively essential speech acts found in a given language. In addition, the target language is often taught as a foreign language, and the students may not be likely to visit or live in a country where the target language is spoken. So, challenges abound.
If we explicitly teach speech acts, are we assuming a level of awareness on the part of students which is out of proportion with the expectations of them in other spheres of language learning? This may well be the case. In addition, we may well ask whether learners strive to be that native-like in their performance (Ishihara, 2003). What if they do not? And what would be the stakes if they stick to their native-language means of performing the given speech acts? These are all issues that arise when efforts are made to have learners be more native-like in their pragmatic behavior.

[ p. 2 ]

Self-access materials on speech acts for learners

Again, at the University of Minnesota, several efforts have been made to support intermediate and advanced language learners in the learning and use of speech acts. For example, a team of authors has generated a students' guide for study abroad with a speech acts component for self-access learning (Paige, Paige, Kappler, Chi, and Lassegard, 2002). In addition, Ishihara and I have constructed a website for learning speech acts in Japanese (apologizing, complimenting, thanking, refusals, and requests) and strategies for appropriate performance: http://www.iles.umn.edu/IntrotoSpeechActs/. What characterizes this website is that it combines both content about the speech acts, which is largely based on empirical studies of what native speakers of Japanese actually do, as well as information as to strategies suggested for performing the speech acts effectively:
  1. Core (i.e., speech act-specific) strategies without which the utterance would not be recognized as the given speech act,
  2. Support strategies that could enhance or expand upon the use of the speech act,
  3. Additional strategies that could also help in the performance of that speech act or related speech acts, but which are not essential,
  4. General strategies that cut across speech acts.
These strategies are color-coded red, blue, black, and green respectively. Evaluational data we have collected from student users of the website would suggest that this color-coding is of help to some learners and not to others. In any event, the following are some examples of strategies for performing speech acts in Japanese:

Numerous unresolved issues

As to be expected with new ventures such as this one many issues are still unresolved. Let us consider three of them.
First, what role does the teacher need to have (if any) if self-access materials are utilized? In particular, is it imperative that the teachers be well-versed in the pragmatics of their language?
Secondly, does a self-access site ensure that learners will have an adequate sense of both how to perceive and to produce pragmatic tone (e.g., contemptuous, mocking, false sincerity, playful, supportive; Beebe and Waring, 2002)? While pre-recorded snippets of language can certainly assist learners in getting a sense of how to use tone, there is no substitute for live interaction to reinforce perceptions.
Finally, how detailed should such a website be? In other words, just how valuable are the insights we gain from sub-sub-categorizing, for example, different ways of showing deference in making requests? Might we not run the risk of discouraging the learners by overloading them with more detail than they can handle?

[ p. 3 ]

Assessing speech acts

Now that we have discussed the teaching of speech acts, let us turn to an even more daunting area, namely, the assessment of what learners have learned about speech acts and how to perform them. We could start by asking, "What is our purpose for assessing speech acts?" In general, the focus has tended to be on assessing speech acts for research purposes. But with the push to teach speech acts in the classroom, what language assessment techniques might we draw on commensurate with this interest? We will give some examples below.
In addition, how much coverage will speech acts receive in a given course? If the coverage is to be minimal, then we might expect that their assessment would be integrated into general proficiency. If it is to be extensive, then presumably assessment would be of achievement with that material.
The following are some of the challenges associated with attempt to assess classroom speech act performance:

Some suggested areas for assessment: What to assess

We could start our assessment of speech acts by checking to see the extent to which learners are able to use speech act-specific core strategies — the set of strategies taught in the material; those that can "easily" be observed and measured (e.g., for apology — expression of apology, acknowledgment of responsibility, explanation/excuse, offer of repair, promise of non-recurrence). In addition we could check to see how much control they appear to have over both the sociopragmatic factors (i.e., whether the speech act can be applied in a given situation) and pragmalinguistic factors (i.e., the language forms that are appropriate in the given situation).
Finally, we could look to see whether the learners are able to deploy appropriate modifications of the speech act, such as:

How to assess speech acts

Much to the benefit of practitioners in the field, researchers at the University of Hawai'i have provided a suggested series of measures of pragmatic ability (requests, refusals, and apologies) along a continuum from free to cued (Hudson, Detmer, and Brown, 1994). Others have benefited from these suggestions by constructing assessment batteries making use of them. For example, Yamashita (1996) developed a battery for assessing speech act ability in Japanese as a foreign language and Yoshitake (1997) did the same for EFL learners. The battery included an oral discourse completion task (DCT), a written DCT, a video-taped role play (3 speech acts), self-assessment (speculation; rating own performance on a video), and a multiple-choice task. For greater validity, it is probably best to use multiple measures in order to approximate the respondents' genuine abilities, as reflected by the Yamashita and Yoshitake batteries.

[ p. 4 ]

As with any set of measures, there are trade-offs associated with using more open as opposed to more closed types of assessment. Open role play, for example, allows for the full operation of turn-taking, sequencing of moves, and negotiation of meaning (Kasper and Dahl, 1991: 228-9). Written response, on the other hand, may foster more thoughtful responses, possibly more indicative of a speaker's competence.
Let us take a look at some actual item types, first looking at the perception of speech acts, and then at the production of them — first by means of the classical DCT and then by means of a multiple-rejoinder DCT:

Perception of a Speech Act

Rate the following responses according to whether they are "acceptable," "more or less acceptable," or "unacceptable" in an American English situation: 
(1) A student forgets to return a book to a professor. Student:

	a. ____ Oh, damn!  I forgot it.
	b. ____ Sorry.  I forgot.
	c. ____ Oh, I'm really sorry.  I completely forgot.
	d. ____ Oh, well, I've had a lot on my mind lately.

(2) A young woman bumps into your shopping cart at the supermarket and 
    some of your groceries spill onto the floor.  
    Aside from helping you pick them up, she says:

	a. ____ Sorry.
	b. ____ Please forgive me.
	c. ____ I'm very sorry.
	d. ____ I'm really sorry.

Production of a Speech Act: Classical DCT

(1) You promised you'd buy your neighbor medicine for her sick child
    while in town, but you forgot.
      
    Your neighbor: "Did you get the medicine?" 
     
    You: _________________________

(2) You don't stop in time at a red light and bump into the car 
    in front of you.  The other driver and you get out and see that there is 
    damage to the other car. The other driver is very upset.
    
    You: _________________________

Production of a Speech Act: Multiple-Rejoinder DCT

(1) You find a bargain air ticket to a city where you have great friends.  
      In order to take advantage of this deal, you need to ask your instructor 
      for an extension on a paper that you were supposed to hand in after the weekend. 
    
    You: __________________________

    Professor: Well, you know, you had plenty of time to work on this paper 
    already. There was no need to wait until the last minute to prepare it.  

    You: __________________________ 

    Professor:  I'm sorry, but I can't really agree to give you an extension
    on this paper.  I don't think that going to visit some friends during the 
    semester is a good enough reason for an extension.

    You:  _________________________

    Professor:  Well, I'm not so thrilled about doing it.  It's not my policy.

    You:  _________________________

    Professor:  Ok, well, just this time.

[ p. 5 ]

Taking a closer look at multiple-rejoinder discourse completion

The multiple-rejoinder approach is meant to get more data than the one-shot approach (see Cohen & Shively, 2002/2003). What are the reactive effects of the measure? To find out, we could have raters assess how well each rejoinder fits into the discourse (say, on a scale of 1-5). We could have the raters look at the following factors:

So how do you actually rate oral and written speech act production?

The construction of a rating scale and even the calibration of the ratings themselves does not ensure that certain issues of reliability and validity have been adequately handled. The following are some of the issues that can come up. One is whether we use predetermined criteria for each speech act — that is to say, those criteria which reflect what was taught in a particular speech act unit. If the assessment is meant to be criterion-referenced, then this might be a good idea. If it is intended to be more a measure of general proficiency, then perhaps it would not be so desirable to predetermine appropriate response.
Following up on this issue of predetermined scoring, do the teacher or raters determine their own intuitive coding holistically and then give their rationale for it? In such a case, for instance, raters would be asked to take notes on what is inappropriate and incorrect for each rejoinder within a given vignette. Furthermore, would the responses be anchored to baseline data from native speakers for each criterion?
And what if we want to assess gain? For a research study, we could scramble the pre-and posttest data for all learners on each speech act, so that raters would not know which was which. But in so doing, we would lose the ability to do a fine-tuned analysis of pre- and post-performance. Might it be better for raters to be comparing the pre- and posttest data learner-by-learner? And are the raters told in advance which were the responses from pretesting and which were from posttesting? And finally, is each rejoinder scored separately or are all rejoinders scored together as if they were one single turn? In our work at the University of Minnesota, we have found it preferable to score all the rejoinders together as one response.

Discussion

". . . the best approach is to collect data from more than one measure and all the same, to view these measures as an approximation of the respondent's speech act ability."
While researchers will continue to fine-tune our descriptions of speech acts, there appears to be sufficient information for teachers and language assessment specialists to develop measures of speech act ability. As this JALT Pan-SIG plenary indicated, there exist numerous formats to choose from, for assessing both the perception and the production of speech acts.

[ p. 6 ]

Since pragmatic behavior by its very nature varies, the best approach is to collect data from more than one measure and all the same, to view these measures as an approximation of the respondent's speech act ability. In fact, might it not be a good idea to use a portfolio approach? While pre-existing measures of speech act performance may be appropriate in a given L2 classroom, designing speech act tasks that really "capture" the characteristics and interests of the given group of L2 students may be preferable. It may be said that language assessment suffers a bit too much from the "one-size-fits-all" mentality. While it may be expedient to use pre-existing measures of pragmatic ability, it may be more informative and also more acceptable to the learners themselves if the measures genuinely reflect the material that they were exposed to and is sensitive to the issues that were covered in their instructional materials.

References

Bardovi-Harlig, K. & Mahan-Taylor, R. (Eds.). (2003). Teaching pragmatics. Washington, DC: US State Department. Retrieved on Feb 12, 2004 from http://exchanges.state.gov/education/engteaching/pragmatics.htm.

Beebe, L. M. & Waring, H. Z. (2001, February 3). Sociopragmatic vs. pragmalinguistic failure: How useful is the distinction? Paper presented at the NYSTESOL Applied Linguistics Winter Conference, CUNY Graduate Center.

Beebe, L. M & Waring, H. Z. (2002, April 7). The pragmatics in the interlanguage pragmatics research agenda: The case of tone. Paper presented at the AAAL Colloquium entitled, "Revisioning Interlanguage Pragmatics Research," Salt Lake City.

Blum-Kulka, S., & Olshtain, E. (1986). Too many words: Length of utterance and pragmatic failure. Studies in Second Language Acquisition, 8 (1), 47-61.

Cohen, A. D. & Ishihara, N. (2004). A Web-Based Approach to Strategic Learning of Speech Acts. Report to the Center for Advanced Research on Language Acquisition (CARLA). Retrieved on Feb 12, 2004 from http://www.carla.umn.edu/speechacts/Speech_Act_Project_Rept.pdf.

Cohen, A. D. & Shively, R. L. (2002/2003). Measuring speech acts with multiple rejoinder DCT's. Language Testing Update, 32, 39-42.

Paige, R. M., Cohen, A. D., Kappler, B., Chi, J. C., & Lassegard, J. P. (2002). Maximizing study abroad: A students' guide to strategies for language and culture learning and use. Minneapolis, MN: Center for Advanced Research on Language Acquisition.

Roever, C. (2004). Difficulty and practicality in tests of interlanguage pragmatics. In B. Boxer & A. D. Cohen (Eds.), Study speaking to inform second language learning (pp. 283-301). Clevedon, England: Multilingual Matters.

Rose, K. R. & Kasper, G. (Eds.) (2001). Pragmatics in language teaching. Cambridge, UK: Cambridge University Press.

Thomas, J. (1983). Cross-cultural pragmatic failure. Applied Linguistics, 4, 91-112.

Varghese, M. & Billmyer, K. (1996) Investigating the structure of discourse completion tests. Working Papers in Educational Linguistics 12 (1), 39-58.

Yamashita, S. O. (1996). Six measures of JSL pragmatics. (Technical Report #14). Honolulu, HI: Second Language Teaching & Curriculum Center, University of Hawaii at Manoa.

Yoshitake, S. (1997). Interlanguage competence of Japanese students of English: A multi-test framework evaluation. Unpublished doctoral dissertation, Columbia Pacific University, San Rafael, CA.

Yule, G. 1996. Pragmatics. Oxford, UK: Oxford University Press.

See Cohen and Ishihara (2004) for more on the strategies component of the website.

2004 Pan SIG-Proceedings: Topic Index Author Index Page Index Title Index Main Index
Complete Pan SIG-Proceedings: Topic Index Author Index Page Index Title Index Main Index

[ p. 7 ]

Last Next