'' Measuring Productive Vocabulary in English Language Learners''
Abstract
Key words:
academic vocabulary; ESL writing; L1 literacy, L2 reading, vocabulary
assessment
1 Introduction
Achieving a competent level in college writing
is a hurdle that many students must overcome when they begin their college
careers. This is especially true for ESL students whose first language
literacy skills range over a wide scale. As writing instructors, we
strive to come up with a winning formula for our students. In doing this,
we try to find the right balance of what seem to be the obvious tools for good
writing. One of these tools is vocabulary.
In order to help students build on their
vocabulary, we need to have a way of assessing their lexical knowledge.
This might seem at first to be a straightforward task. But if the vast
amount of literature on the subject doesn’t dispel the misconception, we need
only ask ourselves the following question: “What exactly does it mean to know a
word?” Is it enough to know its definition? It would make
sense that to claim knowledge of a word, one should know its meaning - but
which meaning? Polysemy is a very common feature of English words.
Furthermore, a person can look up a word’s definition and perhaps not know how
to use the word in a sentence. Thus, it must also be necessary to know a
word’s syntactic behavior (e.g., part of speech, phrase structure), its
semantic properties (e.g., male/female, animate/inanimate), and its potential
semantic contexts. There are numerous other requirements that might be
considered, such as morphological properties (e.g., plural form),
pronunciation, and spelling. According to Bachman and Palmer (1996), knowledge
of vocabulary is part of grammatical knowledge that involves knowledge of
syntax as well as phonology and orthography. Clearly, this is a complex
and somewhat controversial issue. Thus, it is critical for anyone
involved in the assessment of lexical competence to establish a consistent set
of criteria for determining word knowledge.
When speaking of vocabulary in the context of
writing, it is necessary to acknowledge the receptive/productive dichotomy of
lexical knowledge. In simple terms, “receptive” or “passive” vocabulary
generally refers to the ability to read or hear a word and understand the word
in that context. “Productive” or “active” vocabulary is needed in order
to speak and write. This distinction, which is much more complex, particularly
with respect to assessment, will be discussed in this paper.
The paper also reports on data from a recent
study (Sepp & Morvay, 2012) examining, among other things, the relationship
between vocabulary knowledge, reading, and writing proficiency. The
research involved the creation of a series of three tests for vocabulary
measurement. Taking into consideration two contrasting perspectives on the role
of vocabulary in language assessment, the investigators measured whether
learners know the meaning and usage of a set of words, taken as independent
semantic units. This was done by means of a multiple choice test and a cloze
test. The third assessment instrument was an essay prompt, designed to
assess their lexical ability in the context of a language-use task, namely the
productive skill of writing.
In measuring the productive vocabulary of ESL
learners for this research, the authors decided to focus on a certain type of
vocabulary, more precisely, academic vocabulary. Rather than choose the
target words randomly, a popular and highly regarded resource known as the
Academic Word List (AWL) by Coxhead (2000) was used. This list contains 570
word families from a 3.5 million word academic corpus. The AWL includes
words occurring at least 100 times in the corpus that are not part of the
General Service List (GSL) (West 1953), a list of roughly 2,000 of the most
commonly used words in English. According to Schmitt (2010: 79), “the AWL is
the best list of academic vocabulary currently available, and is widely used in
vocabulary research” .
Data from the study were examined statistically,
using Pearson Correlations. The findings, while suggesting a weaker than
expected correlation between vocabulary and writing competence, underscore the
complex nature of assessing each of these skills and the need for research in
this area.
2 Research Summary
In Sepp & Morvay (2012), students at an
urban community college were tested in academic vocabulary and morphosyntactic
knowledge. The test results were compared to their performance on standardized
reading and writing tests, which determine whether they are ready for
credit-bearing college composition classes. The discussion here revisits
the vocabulary data produced from the earlier study, exploring a few additional
variables such as word count and TTR.
The vocabulary tests were administered to 95
students, who were enrolled in an intensive writing course for advanced ESL
students during the spring 2011 semester. Seven advanced ESL writing classes
participated in the study. Each class consisted of approximately 25
students from a variety of linguistic backgrounds. The statistical
analyses presented here are based on the entire group of 95 students and also a
subset of this group characterized by a higher word count on the essay task.
One goal of the research was to ascertain the
most effective way to measure the productive vocabulary of ESL students in
advanced developmental writing courses. To that end, the investigators compared
the results of the three test types in order to determine whether there was a
significant correlation between any of the tests and the outcome of a
standardized writing test, in this case, the CUNY Assessment Test in Writing
(CATW). Students are required to pass the CATW in order to exit the
college’s remedial writing course sequence, so it is a high-stakes test. It is
also a formidable challenge for many of these students. The CATW is a
reading-based essay test and it is scored analytically for content, structure, and
language use. Since reading is a component of the writing exam, we also
looked for correlations between vocabulary and reading in the dataset.
2.1 Test Design
The context for the multiple choice and cloze
tests was derived from three articles on three different topics (see Appendix),
which were adapted for assessment of the target skills. Thirty target
words (10 per topic) from the Academic Word List (AWL) developed by Coxhead
(2000) were used in the vocabulary tests.
Three tests were created to assess academic
vocabulary:
i.
Multiple Choice Vocabulary (Test 1) - 10 target words
were tested in a multiple choice format with a 15-minute time constraint.
Students were given three choices per item. The non-target words selected
for the tests were intended to give students pause but not to confuse
them. In each case, the target word should be unambiguously correct.
e.g., During the first few years of life,
it's (crucial/ trivial/ decisive) to meet a child's nutritional needs.(from
Test 1, topic A – see Appendix)
ii.
Cloze Vocabulary (Test 2) – the same 10 target words in Test
1 were deleted from a passage (except for the first letter) where students had
to provide the word under a 20-minute time constraint. Only the first
letter was provided because providing more than that made the task more of a
receptive task than a productive one.
e.g., During the first few years of
life, it's c
to meet a child's nutritional needs in order to e
proper growth and also to e
a lifelong habit of healthy eating.(from
Test 2, topic A– see Appendix)
iii.
Essay - students were instructed to write a
60-minute timed essay on a specified topic and to incorporate 10 target
vocabulary items into their writing. The topics were of a fairly general
nature and designed to mimic the theme of the texts used in the other two tests
in order to reasonably elicit the target words. For example, students were
asked to try to use words such as portion, ensure, and crucial in
their response to the prompt below.
Essay prompt A
It is common knowledge
that childhood obesity is a problem in this country. What can parents or
educators do to help end this problem?
Testing took place between week 3 and week 11 of
a 15-week semester. All tests were administered in the classroom. The two
vocabulary tests that were given in weeks 3 and 5 were administered in a
randomized order between the various classes; in other words, some classes were
given Test 1 and then Test 2 and others were given Test 2 and then Test
1. The essay was administered during week 11.
A total of 130 students completed the essay
task. The average length was 330 words and the topic didn’t seem to have any
effect on how much students wrote, as averages were virtually the same for all
three topics.
Tests were scored by the co-investigators and
then data were compiled. To ensure accuracy and consistency, scoring of all
tests was cross-checked by the two co-investigators. Test results from a
set of 80 were used in the original data analysis. But since there were 95
students who completed all three vocabulary tasks, data were reanalyzed to
include the additional 15 cases. In addition, two other variables
were considered: essay length (word count) and vocabulary range
(type-token ratio). Thus, a total of 5 vocabulary variables were used in
the statistical analysis. These are identified and defined below:
i. MultiV
– percent correct on multiple choice vocabulary test
ii. ClozeV –
percent correct on cloze vocabulary test
iii. EssayV – percentage of 10
target words student was able to use correctly at least once
iv. TTR – type-token
ratio per essay
v. Word
count - total number of tokens per essay
2.2 Scoring Criteria
Scoring the multiple choice tests was fairly simple. For the cloze
vocabulary test, the first letter of the word was provided, so in many cases
there were a number of non-target words that fit the context as well.
Such words were accepted as correct. In addition, while the response in
general should match the syntactic category of the target word, if an adverb
form were used instead of an adjective – e.g., proportionately instead
of proportionate – and it fit both the structural context and the
meaning, then it was accepted. Errors in tense and/or number were
tolerated, as were minor spelling errors. The criteria for scoring the
essay were similar to those used for the cloze tests:
i.
Scores
were calculated as the percentage of the 10 target words the writer was able to
use correctly (using each correctly once was sufficient).
ii.
Correct
responses had to be globally and locally plausible. The vocabulary item
also had to be written in the form of the correct syntactic category, with the
exception of adjectives and adverbs, in which case either form was accepted as
long as it was deemed appropriate to the context. Minor spelling errors
and errors in tense and/or number were tolerated.
iii. Credit was given for
target academic words only.
The actual scoring of the essay was of course a more complicated
process due to the range of contexts that had to be considered, as well as
differences in students’ syntactic ability.
2.3 Analysis of Data
Average test scores were compared vis-à-vis both
developmental writing and reading competence. Reading competence is taken
as the numeric score attained on the university’s standardized reading test.
Writing competence was determined based on whether the student passed the
writing test at the end of the semester in which the research was
conducted. Of the 95 participants in this group 24% passed ESL, while 74%
passed reading.
In addition, Pearson Correlations were
calculated to determine the relationship between test scores, vocabulary range,
essay length, and reading and ESL course outcomes.
With regard to average scores, the results (see
Table 1) showed slightly higher average scores on the three vocabulary tests
for students who failed ESL095 at the end of the semester. The essays were used
to calculate two additional measures related to vocabulary size: type-token
ratio (TTR), which measures the range of vocabulary used in a text, and essay
word count. The average TTR was also higher for students who failed ESL,
while the average word count was considerably higher for passing students at
417:310.
N=95
|
multiV
|
clozeV
|
essayV
|
essay word count
|
essay TTR
|
Passed ESL
|
0.73
|
0.28
|
0.29
|
417
|
0.45
|
Failed ESL
|
0.75
|
0.34
|
0.32
|
310
|
0.49
|
All students
|
0.75
|
0.33
|
0.31
|
336
|
0.48
|
Table 1: Average scores based on ESL outcomes
With respect to reading level, vocabulary scores
were about the same on average in all categories except EssayV, as shown in
Table 2. In this case, students who passed reading performed a bit
better.
N=95
|
multiV
|
clozeV
|
essayV
|
essay word count
|
essay TTR
|
Passed Reading
|
0.74
|
0.33
|
0.32
|
336
|
.48
|
Failed Reading
|
0.75
|
0.34
|
0.29
|
335
|
.48
|
All students
|
0.75
|
0.33
|
0.31
|
336
|
.48
|
Table 2: Average scores based on Reading
outcomes
To gain a more reliable sense of the
relationship between vocabulary and writing / reading competence, Pearson
Correlations were computed. This was done for the set of 95 and also for
a subset of 26, which included data from participants who wrote longer essays
in test 3. Essays in this subset contained 400 or more tokens, and this
dataset is therefore referred to here as “400plus”. Three correlations
tables are presented below, showing the results for: the core set for
vocabulary/writing (Table 3); the 400plus group for vocabulary/writing (Table
4); and the 400plus group for vocabulary/reading (Table 5). Vocabulary/writing
and vocabulary/reading data are provided in separate tables for easier
comparison.
In Table 3, Pearson correlations were calculated
for vocabulary variables, showing a .01 level of significance between clozeV
and multiV. In addition, there were significant correlations at the .05
level between essayV and multiV. Establishing a correlation between
student scores on the three tests shows a relative consistency in the
performance of these assessment instruments.
vocab-eslfinal-95
|
multiV
|
clozeV
|
essayV
|
essay word count
|
TTR
|
ESL final
|
|
multiV
|
Pearson Correlation
|
1
|
.339**
|
.346**
|
-.280**
|
.183
|
-.067
|
Sig. (2-tailed)
|
.001
|
.001
|
.006
|
.076
|
.520
|
||
N
|
95
|
95
|
95
|
95
|
95
|
95
|
|
clozeV
|
Pearson Correlation
|
.339**
|
1
|
.830**
|
-.170
|
.145
|
-.137
|
Sig. (2-tailed)
|
.001
|
.000
|
.099
|
.161
|
.186
|
||
N
|
95
|
95
|
95
|
95
|
95
|
95
|
|
essayV
|
Pearson Correlation
|
.346**
|
.830**
|
1
|
-.120
|
.086
|
-.075
|
Sig. (2-tailed)
|
.001
|
.000
|
.246
|
.410
|
.469
|
||
N
|
95
|
95
|
95
|
95
|
95
|
95
|
|
essay word count
|
Pearson Correlation
|
-.280**
|
-.170
|
-.120
|
1
|
-.705**
|
.385**
|
Sig. (2-tailed)
|
.006
|
.099
|
.246
|
-
|
.000
|
.000
|
|
N
|
95
|
95
|
95
|
95
|
95
|
95
|
|
TTR
|
Pearson Correlation
|
.183
|
.145
|
.086
|
-.705**
|
1
|
-.218*
|
Sig. (2-tailed)
|
.076
|
.161
|
.410
|
.000
|
.034
|
||
N
|
95
|
95
|
95
|
95
|
95
|
95
|
|
ESL final
|
Pearson Correlation
|
-.067
|
-.137
|
-.075
|
.385**
|
-.218*
|
1
|
Sig. (2-tailed)
|
.520
|
.186
|
.469
|
.000
|
.034
|
||
N
|
95
|
95
|
95
|
95
|
95
|
95
|
|
Table 3: Pearson Correlations for vocabulary features and ESL
outcomes – core set
**. Correlation is significant at the 0.01 level (2-tailed).
|
|||||||
*. Correlation is significant at the 0.05 level (2-tailed).
|
|||||||
Vocabulary range (TTR) and essay length are
negatively correlated (-.705) at a .01 level of significance. This
reflects the fact that shorter essays normally repeat fewer words.
There was a significant (p<.01) correlation
between essay length (word count) and ESL095 outcome. But there was no
correlation between the results of the vocabulary tests and the ESL
outcome. Moreover, there was a negative correlation between ESL writing
outcome and TTR. Again, this result is not surprising since many of the
failing essays were rather short. In order to adjust for the length
issue, we decided to analyze a subset of the data which contained essays of 400
or more words. Pearson Correlations were recalculated and the same
pattern emerged. The correlations are represented in Table 4 below.
400plus
N=26
|
ESL Final
|
multiV
|
-.324
|
clozeV
|
-.076
|
essayV
|
-.170
|
essay word count
|
.459*
|
TTR
|
.079
|
Table 4: Pearson Correlations for vocabulary
features and
ESL outcome – 400plus
*. Correlation is significant at the 0.05 level
(2-tailed)
The data revealed a significant correlation at
the .05 level between essay length and ESL writing outcome but no correlation between
range of vocabulary and ESL outcome. The 400plus essays were also
analyzed based on reading ability (Table 5), and while essay length showed no
correlation here, the correlation between range of vocabulary (TTR) and reading
was significant at the .05 level.
400plus
N=26
|
Reading P/F
|
multiV
|
.196
|
clozeV
|
.021
|
essayV
|
-.028
|
essay word count
|
-.175
|
TTR
|
.408*
|
Table
5: Pearson Correlations for vocabulary features and
reading
outcome – 400plus dataset
3 Discussion and Conclusions
The lack of correlation between students’
performance on the three vocabulary tests and writing outcomes suggests that
targeting a specific set of vocabulary items may not be an effective way to
approximate students’ lexical competence. Furthermore, using the AWL as a
basis for testing vocabulary ability may be too limiting. While it is a valued
resource in present-day research, it derives from a corpus of edited academic
texts, and thus may not be the best indicator of a non-native speaker’s
vocabulary size. Also, the Academic Word List is made up of high
frequency academic words, and good writers may simply opt for less frequent but
equally appropriate words than those in the AWL. In other words, one can
potentially demonstrate a good command of vocabulary without using words from
the AWL.
The fact that reading correlated to a broader
vocabulary is not surprising. Likewise, the connection between relatively
longer essays and better writing skills also makes sense. On the other
hand, the notion that a broader vocabulary does not necessarily correspond to
competent writing, even in the 400plus dataset, may seem
counterintuitive. In fact, a previous study conducted by one of the
authors (Sepp 2010) revealed that TTR (vocabulary range) did significantly correlate
(p<.05) to a positive outcome in ESL writing. However, the latter
research was based on a different standardized essay test (CUNY ACT), which was
scored holistically. The CATW is scored analytically, meaning that
readers assign individual scores for different aspects of structure, content
and language use. Language use is scored for sentence variety, vocabulary, and
sentence mechanics. Thus, it might be surmised that readers who are
scoring analytically pay more attention to clarity of expression than range of
vocabulary. This, however, is a topic for another study.
On the other hand, the results might also be
different with a larger dataset. There is no doubt that better writers tend to
have a better vocabulary, but the question here, from a pedagogical
perspective, is how much emphasis should be placed on vocabulary instruction in
order to help ELLs reach a level of acceptable competence. What makes a
college student’s writing good enough?
In the end, the relationship between vocabulary
and competent writing may be intrinsically linked to what the assessors are
looking for and how “competence” is perceived.
Authors:
Gabriella Morvay, Ph.D.
Assistant Professor of Linguistics and ESL
Borough of Manhattan Community College
City University of New York
E-mail: gmorvay@bmcc.cuny.edu
Mary Sepp, Ph.D.
Assistant Professor of Linguistics and ESL
Borough of Manhattan Community College
City University of New York
E-mail: msepp@bmcc.cuny.edu
References
Coxhead, A.
(1998). An academic word list (English Language Institute Occasional
Publication No. 18). Wellington, NZ: Victoria University of
Wellington.
Coxhead, A. (2000).
A new Academic Word List. TESOL Quarterly 34 (2), 213-38.
Fitzpatrick, T (2007).
Productive vocabulary tests and concurrent validity. In H. Daller, J.
Milton, and J. Treffers-Daller (eds.), Modelling and Assessing
Vocabulary Knowledge (pp. 116-132). Cambridge: Cambridge University
Press.
Fitzpatrick T. &
Clenton, J. (2010). The challenge of validation: Assessing the
performance of a test of productive vocabulary. Language Testing,
27(4), 537-554.
Henriksen, B.
(1996). Semantisation, retention and accessibility: Key concepts in
vocabulary learning. Paper presented at the AILA Congress, Jyvaskyla,
Finland. August 1996.
Faerch, K. Haastrup, K.
&Phillipson. (1984). Learner Language and Language Learning.
Copenhagen: Multilingual Matters.
Laufer, B. (1998).
The development of passive and active vocabulary in a second language: same or
different? Applied Linguistics , 19, 255-271.
Laufer, B. &Goldstein,
Z. (2004). Testing vocabulary knowledge: Size, strength, and computer
adaptiveness. Language Learning, 54, 399-436.
Laufer,
B. & Nation, P. (1999). A vocabulary-size test of
controlled productive ability. Language Testing, 16, 33-51.
Lado, R.
(1961). Language Testing. London: Longman.
Meara, P. (1990).
A note on passive vocabulary. Second Language Research, 6
(2). 150-154.
Meara, P. &
Fitzpatrick, T. (2000). Lex30: an improved method of assessing productive
vocabulary in an L2. System, 28, 19-30.
Melka, F. (1997).
Receptive vs. productive aspects of vocabulary. In N. Schmitt and M.
McCarthy (eds.), Vocabulary: Description, Acquisition, and Pedagogy
(pp. 84-102). Cambridge: Cambridge University Press.
Nadarajan, S.
(2008).Assessing in-depth vocabulary ability of adult ESL learners. The
International Journal of Language, Society and Culture, 26, 93-106.
Nation, I. S. P.
(1983). Testing and teaching vocabulary. Guidelines, 12-25.
Nation,
I.S.P. (1990). Teaching and Learning Vocabulary. Boston, MA:
Heinle and Heinle.
Nation, I. S. P.
(2001). Learning vocabulary in another language. Cambridge:
Cambridge University Press.
Nation, I.S.P.
(2006). How large a vocabulary is needed for reading and listening? The
Canadian Modern Language Review, 63,1, 59-82.
Palmberg, R.
(1987). Patterns of vocabulary development in foreign language
learners. Studies in Second Language Acquisition, 9, 201-220.
Pearson, P. D., Hiebert,
E. H., and Kamil, M. L. (2007). Reading Research Quarterly, vol.
42, no. 2, 282-296.
Pigott, P. (1981).
Vocabulary growth in EFL beginners. MA project, Birkbeck College, London,
England.
Read, J. (2000). Assessing
Vocabulary. Cambridge: Cambridge University Press.
Read, J.: (2007).
Second language vocabulary assessment: Current practices and new
directions. International Journal of English Studies, vol. 7 (2),
105-125.
Sepp, M. (2010). Getting
Assessment Right. The Inquirer, fall 2010..BMCC, CUNY
Sepp, M. & Morvay,
G. (2012). Productive Vocabulary, Morphosyntactic Knowledge, Reading Ability,
and ESL Writing Success. Iranian Journal of TEFLL, vol. 2 (2), 3-22.
Thorndike, E. &
Lorge, I. (1944). The teacher’s word book of 30,000 words.
New York: Teachers CollegePress.
Wesche, M. & Paribakht,
T. S. (1996). Assessing second language vocabulary knowledge: depth vs.
breadth. Canadian Modern Language Review 53, 13-39.
West,
M. (1953). A General Service List of English words. London:
Longman, Green.
Wilson,
R. & Beineke, L. (1979). Applications of graph theory.
New York: Academic Press.
Xue,
G. & Nation, I.S.P. (1984). A university word list. Language Learning
and Communication, 3, 15-29.
Link :
https://sites.google.com/site/linguisticsandlanguageteaching/home-1/volume-4-2013-issue-1/volume-4- 2013-issue-1---article-morvay-sepp
ARTICLE
RIVIEW
A . DEMOGRAPY
§ Title : Measuring Productive
Vocabulary in English Language Learners
§ Author
: Gabriella Morvay, Ph.D.
Assistant Professor of
Linguistics and ESL
Borough of Manhattan
Community College
City University of New
York
Mary Sepp, Ph.D.
Assistant Professor of
Linguistics and ESL
Borough of Manhattan
Community College
City University of New
York
E-mail: msepp@bmcc.cuny.edu
§ Number
of chapter : 1 – 22 pages
B . BACKGROUND
§ Purpose : examine the most effective way to measure
productive vocabulary in order to determine its role in L2 writing. For
pedagogical purposes, the question of
whether it is a worthwhile endeavor to invest valuable instruction time into
teaching academic words to ESL community college students. These goals were
accomplished by testing students’ knowledge of academic vocabulary and
calculating to what extent this knowledge relates to competent writing, i.e.,
what is deemed competent according to established criteria.
C . CONTENT
§ students
at an urban community college were tested in academic vocabulary and
morphosyntactic knowledge. The test results were compared to their performance
on standardized reading and writing tests, One goal of the research was to
ascertain the most effective way to measure the productive vocabulary of ESL
students in advanced developmental writing courses.
To that end, the investigators
cificant between any of the tests and the outcome of a standardized writing
test, in this case, the CUNY Assessment Test in Writing (CATW). Students are required to pass the CATW in
order to exit the college’s remedial writing course sequence, so it is a
high-stakes test.
It is also a formidable challenge
for many of these students. The CATW is
a reading-based essay test and it is scored analytically for content,
structure, and language use. Since
reading is a component of the writing exam.
D . STRENGTHNESS AND WEAKNESS
§ Strengthness:
-Abstrak clear, so
that by reading the abstract alone readers know the
results of these studies
-kesimpulan made already detailed and clearly explained
-procedure research compiled regularly, so it is easy to understand
-kesimpulan made already detailed and clearly explained
-procedure research compiled regularly, so it is easy to understand
§ Weakness
:
-No response from the
public about the results of these studies
-No percentage
-No suggestions for future research
-No percentage
-No suggestions for future research
E . REMARKS OR CONCLUTION
§ performance
on the three vocabulary tests and writing outcomes suggests that targeting a
specific set of vocabulary items may not be an effective way to approximate
students’ lexical competence.
Furthermore, using the AWL as a basis for testing vocabulary ability may
be too limiting. The fact that reading to a broader vocabulary is not
surprising. Likewise, the connection
between relatively longer essays and better writing skills also makes
sense. On the other hand, the notion
that a broader vocabulary does not necessarily correspond to competent writing.
