Reading 2 - test specification for writing test - vstep
- 1. Test specifications for Writing test - VSTEP 1. General information: Time: 60 minutes General description of the test: the test which is composed of 2 parts aims at assessing students’ written communicative competence. It targets at the test takers at level 3 to level 5 CEFR-VN levels. Task demands and response format: o Input: in written form, with the topic written in the target language at level 3 of VSTEP. The topics are familar. o Expected output: a letter serving different purposes of communication (task 1) and an essay giving information, expressing and defending opinions (task 2). Description of each task: o Task 1: Time limit: 20 minutes. Length: 120 words at least. Weighting: 10/30. Test takers write a(n) letter/ email in response to a(n) given letter/email, or test takers write a(n) letter/email serving different communication purposes. This task tests written interaction skills. o Task 2: Time limit: 40 minutes. Length: 250 words at least. Weighting: 20/30. Test takers write an essay on a given topic, using his/her experience and knowledge to support his/her arguments. This task tests written production skills. Number of items: 2 Score reporting plan: A raw score out of 30 is converted to a score in 10-point scale
- 2. Task demands and response format Time recomme nd-ation Intended operations Task types Instructions Task 1 Input: Part of a(n) letter/email about an event (80-90 words) related to a familiar, everyday situation. or A familiar, everyday situation (80-90 words) that requires correspondence Expected output: A (reply) letter/email (at least 120 words) Content knowledge: familiar matters regularly encountered in work, school, leisure; topics which are familiar, or of personal interest. 20 minutes Can convey information and ideas on abstract as well as concrete topics (e.g work, study or films, music), check information and ask about or explain problems with reasonable precision. Can write personal letters describing experiences, feelings and events in some detail. a(n) letter/ email in response to a(n) given letter/ email a(n) letter/ email asking for information, describing experiences, feelings and events. Test takers are asked to write a(n) letter/ email in response to a(n) given letter/email or to give/ask for information.
- 3. Task 2 Input: A given text/an extract from a book/an article on a given topic (50-60 words). The topics may be related to Vietnamese context. Output: (at least 250 words) An essay evaluating/giving opinions An essay explaining advantages/disadvantages An essay explaining causes/effects An essay explainin problems/solutions Content knowledge: Education, health, entertainment, transportation, history, art, etc 40 minutes Can write an essay supporting arguments, highlight main ideas and supporting them with relevant details. Can write an essay to develop an idea, giving reasons for arguments, explaining the advantages and disadvantages. An essay Test takers are asked to write an essay on a given topic, using his/her experience and knowledge to support his/her arguments.
- 4. 2. Sample instructionand items: Task 1: You should spend about 20 minutes on this task. You have invited an American businessman, Mr. Noah Watson, to give a talk at your English club monthly meeting. Read part of the letter from him below. I would be glad to come and give a talk to your English club. In order to make a good preparation, I would like some information about my audience. I could talk about either my experience when starting up my company or some general rules when doing business. Which one would be more interesting? Also, I am staying in a hotel in the city centre. Could you tell me where the meeting is and how to get there? I look forward to meeting you soon. Write a letter responding to Mr. Noah Watson. You should write at least 120 words. You are not allowed to include your name. Your response will be evaluated in terms of Task fulfilment, Organization, Vocabulary and Grammar. Task 2: You should spend about 40 minutes on this task. Read the following extract from a book chapter in 1 minute: Read the following extract from a book chapter in 1 minute: Now, write an essay in 40 minutes to discuss what measures in your opinion are more effective in controlling the crime rate. Give reasons for your answer and include any relevant examples from your own knowledge or experience. Your response will be evaluated in terms of Task fulfilment, Organization, Vocabulary and Grammar. You should write at least 250 words. There are fewer violent crimes in some countries in recent years, but the overall number of crimes is still very high. People have different opinions about the best way to control the crime rate. Some suggest preventing people from an/ any illegal action before they do it while others prefer punishing criminals more heavily. Adapted from Academic Listening Encounters - Life in Society, Chapter 8, Page 113-114
- 5. Part 2: Guidelines for the design of effective writing prompts 2.1. General principles indesigning effective writing prompts 2.1.1. Clearly define the intended writing skills to be tested by the item. Decide which of the following intended writing skill lends itself better to be assessed by an essay question. Example A You should spend 40 minutes on this task. Read the following text from a book about tourism. Tourism has become one of the fastest growing industries in the world. Millions of people today are travelling farther and farther, not only in the summer but all year round. Some people argue that the development of tourism has had negative effects on local communities; others think that its influences are positive. Express your opinion on the issue. Include reasons and relevant examples to support your answer. Example B You should spend 40 minutes on this task. Read the following text from a book about tourism. Tourism has become one of the fastest growing industries in the world. Millions of people today are travelling farther and farther, not only in the summer but all year round. Some people argue that the development of tourism has had negative effects on local communities; others think that its influences are positive. Write an essay to discuss the effects of tourism on the tourist destinations. Include reasons and relevant examples to support your answer. Example B is more useful for guiding the development of an essay question. Specific intended writing skills are crucial to designing effective writing questions. If the expected outcome to be assessed lacks clarity and specificity, the writing question meant to assess students’ performance of a skill will likely assess something other than what is intended. Example A is too general to provide clear guidance in answering the question. The directive verb, “express” is vague and difficult to assess. It is especially unclear about what form of the answer should be. Meanwhile, example B explicitly requires candidates “to write” and the desired form of the answer is “an essay”. In specifying the intended writing skills, item writers should clarify the performance that students should be able to demonstrate as a result of what they have learned. Tested writing skills meant to guide the development of test items will typically begin with a specific directive verb. The statement will describe the observable behavior, action or outcome that students should demonstrate.
- 6. 2.1.2. Clearly define and situate the task within a problem situation. a. Define the task With some writing questions, test takers can feel like they have an infinite supply of lead to write a response on an indefinite number of pages about whatever they feel happy to write about. This can happen when the writing question is vague or open to numerous interpretations. Remember that effective writing questions: provide candidates with an indication of the types of thinking and content to use in responding to the question. A clearly defined task will specify the performance test takers should exhibit when responding to the writing question. A clearly defined task is composed of a directive verb and the object of that verb. For example, consider the following tasks: Limiting the subject matter for a given question depends on different factors. Among other things, how to restrict the task can depend on the hierarchy of knowledge related to the question. The following example illustrates the process of limiting the scope of the task for a given essay question. Example of an Evolving Essay Question that Becomes More Focused 1. Less focused essay question: Evaluate the impact of the social networking sites. 2. More focused essay question: Evaluate the impact of the social networking sites on the young generation. Failure to establish adequate and effective limits for the response to the writing question allows test takers to set their own boundaries for their response, meaning that they might provide responses that are outside of the intended task, that are too long, or that only address a part of the intended task. Test takers’ failure to answer within the intended limits of the essay question can be ascribed to poor or ineffective wording of the task to some extent. Therefore, item writers are responsible for writing questions in such a way that they provide test takers with clear boundaries for responses. In short, item writers should avoid indeterminate questions. One way to avoid indeterminate questions is to stay away from vocabulary that is ambiguous. Moreover, item writers should also avoid including vocabulary that is too advanced for the test takers. The suggested language level for both tasks are B1 (CEFR) vocabulary which may account for 80-90% of the words used in the prompts. b. Create a problem situation
- 7. Once the writing task is clearly defined and the content that test takers are to use in accomplishing the task has been delimited, the question is ready to be situated in a problem. Both the tasks and the problems are key elements of writing questions. The problem in writing questions includes the unsettled matter or undesirable state of affairs that needs to be resolved. The purpose of the problem is to provide the students with a context within which they can demonstrate the writing performance to be assessed. Writing problem situations requires creativity and clarity. The circumstances should be clear in the sense that even a minimum test taker of level 3 can identify the described event and its related issues . At the same time, the problem situation should not be completely foreign to the test takers’ experience. Example A: Analyze the task and the problem of the following writing question. You should spend 20 minutes on this task You received an email from your English pen friend, Jane, to ask information about one of your friends. Read part of her email below. ...I’ve just got an email from your friend, Mai. She is going to do a course in London t his summer and wants to rent a room in my house. Could you tell me a bit about her (work/study, personality, hobbies and interests) for me to see if she would fit in with my family? … Write an email responding to Jane. You should write at least 120 words. Do NOT include your name. 2.1.3. Present a reasonable task. When defining the task for a writing question, item writers need to make sure that they present a reasonable task to test takers. One of the challenges that item writers face in composing essay questions is that because of their extensive experience and background knowledge they may be tempted to demand unreasonable content expertise from the test takers. Hence, item writers need to make sure that their test takers can “be expected to have adequate material with which to answer the question” (Stalnaker, 1952, p.520). The writing topics should be familiar and accessible to all candidates. Identifying an audience for test takers to write their responses to may also be helpful. This would make the task more authentic. A key element in writing communication is to clearly identify and write to a specific audience. 2.1.4. Specify the relative point value, the approximate time limit and word limit in clear directions.
- 8. Specifying the relative point value and the approximate time limit helps test takers allocate their time in answering several writing questions in a writing test because the directions clarify the relative merit of each question. Without such guidelines test takers may feel at a loss as to how much time to spend on a question. When deciding how much time should be spent on a question, the slower candidates can have a clearer plan for their response. Also make sure that test takers can be realistically expected to provide an adequate answer in the given and/or the suggested time. 2.1.5. Specify the assessment criteria. The expected response should be described clearly in terms of the criteria that will be used to judge it. For example, a letter-writing task will be scored without consideration of grammatical accuracy, as long as the response is comprehensible, examinees should be told as much. On the other hand, if an essay will be scored on the basis of grammatical accuracy, vocabulary use, content and organization, those categories should be disclosed to the test takers in the directions. 2.1.6. Improve the essay question through preview and review. The following steps can help item writers improve an item before and after they hand it out to test takers, making the question much more effective. Preview (before handing out the essay question) a. Predict responses. Try to respond to the question from the perspective of a typical candidate. Evaluate whether candidates have the content knowledge and the skills necessary to adequately respond to the question. Repair any possible weaknesses of the essay question before handing out the exam. b. Write a model answer. Before using a question, write model answer(s) or at least an outline of major points that should be included in an answer. Writing the model answer allows reflection on the clarity of the essay question. Furthermore, the model answer(s) serve as a basis for grading responses. Once the model answer has been written compare its alignment with the question and the tested writing skills and make changes as needed to assure that the tested writing skills, the essay question, and the model answer are aligned with each other. c. Ask a knowledgeable colleague to critically review the question, the model answer, and the tested writing skills for alignment. Before using the essay question on a test, ask a person knowledgeable in the subject to critically review the essay question, the model answer, and the the tested writing skills to
- 9. determine how well they are aligned with each other. Based the tested writing skills, revise the question as needed. By having someone else look at the test the likelihood of creating effective test items is increased. Try asking a colleague to evaluate the essay questions based on the guidelines for constructing essay questions contained in this guideline. Review (after receiving the responses) After test takers complete the test, carefully review the range of answers given and the manner in which test takers seem to have interpreted the question. Make revisions based on the findings. Writing good questions is a process that requires time and practice. Carefully studying the responses can help evaluate test takers' understanding of the question as well as the effectiveness of the question. 2.2. Principles indesigning writing prompts for Task 1 and Task 2 Guidelines for Task 1 a. Components of an item There are three parts in an item of Task 1: Instruction, Letter extract and Task requirement. Instruction suggests how much time candidates should spend on Task 1 and includes one or two sentences in italics, which often introduce the name of the sender and explain the relationship between the sender and the responder. Letter extract is part of an email or a letter placed in textbox. Task requirement explains the expected output from the candidates, the minimum number of words in the target response, and how candidates’ answers are assessed. The Task Requirement is virtually the same in all items in Task 1, except for the name of the sender. Prompt includes both the instruction in italics and the letter extract. b. Principles in writing Task 1 prompt As Task 1 in the Writing Test aims at examining candidates’ ability in writing correspondence, the expected response is in the form of an email or a letter. When designing questions for Task 1, item writers should select a familiar topic, one which is regularly encountered in work, school or leisure, or of personal interest. The criterionis to make sure that even candidates around Level 3 will not be jeopardized by a less familiar topic. Furthermore, a letter extract needs to be written in a consistent style, using either formal or informal language throughout. This will help to set the tone of the expected response. A letter
- 10. extract in Task 1 has approximately 80-90 words. All fully-developed responses in Task 1 should allow candidates to communicate three main ideas, which means the letter extracts should raise three questions to be answered, or address three problems that need to be settled, or a combination of these. Another requirement is concerned with the complexity of language, including lexis and sentence structures to be used in the instructions and letter extracts. Similar to the requirement about topic familiarity, item writers should design with the level of candidates at the lowest level (around Level 3) in mind. For all prospectus candidates to be able to comprehend the content of a prompt, it is suggested that 80 to 90 percent of the lexis and syntax used throughout the prompt corresponds to Linguistic competence Level 3 descriptors. Item writers, therefore, can refer to Vocabulary English Profile (http://vocabulary.englishprofile.org/) or Cambridge Dictionaries Online (http://dictionary.cambridge.org/) to check the level of a certain vocabulary item. To make it consistent across items in the item bank, a prompt needs to provide information about: (i) the sender; (ii) the relationship between the sender and the responder; (iii) the topic of the letter/email extract; and (iv) the context of the letter/email (why this letter/email is being sent, what is expected from the responder). A prompt also needs to clearly indicate what communication purpose(s) is expected in the response (such as giving direction, asking for/giving advice, making enquiries, thanking, accepting/refusing invitation, …). Guidelines for Task 2 a. Components of an item Similarly to Task 1, there are also three parts in an item of Task 2. Instruction suggests how much time candidates should spend on Task 2 and introduces the genres of the prompts. Prompt is often an extract from a book, part of a magazine article or journal article, etc. Prompts are placed in a textbox. Task requirement indicates the expected response from the candidates and explains what candidates are required to write about. The last paragraph in Task requirement suggests how candidates can support their argument, the minimum number of words in their responses, and lists criteriaagainst which candidates’ answers are assessed. The last paragraph is the same in all items in Task 2. b. Principles in writing Task 2 prompt As Task 2 in the Writing Test aims to assess candidates’ ability of written production, the
- 11. target response is an essay. Topics in Task 2 include more general issues, such as education, health, entertainment, transportation, etc. A prompt in Task 2 should be written in academic language, with the limit of approximately 50 to 60 words. Following the prompt, Task requirement should have 40 to 50 words. Like Task 1, items in Task 2 should use lexis and syntax that accommodate candidates around Level 3. Task requirement (an essay to evaluate/ to give opinion/ to explain advantages and disadvantages/ to discuss problem and solutions …) has to be clearly indicated. Prompts are often written in compound sentences and follow a consistent structure. The first one or two sentence(s) in a prompt needs to introduce the topic and provides the background for a certain issue. The next one or two sentences can serve either of the following functions: to list different opinions about the topic; to introduce a certain problem; or to introduce different aspects of the topic. REFERENCES Andrade, H. and Du, Y. (2005), “Student perspectives on rubric-referencedassessment”, Practical Assessment, Research & Evaluation, Vol. 10 No. 5, available at: http://PAREonline.net/getvn.asp?v¼10&n¼3 (accessed December 14, 2006). Arter, J. and McTighe, J. (2001), Scoring Rubrics in the Classroom, Corwin, Thousand Oaks, CA. Boulet, J.R., Rebbecchi, T.A., Denton, E.C., Mckinley, D. and Whelan, G.P. (2004), “Assessing the written communication skills of medical school graduates”, Advances in Health Sciences Education, Vol. 9, pp. 47-60. Bresciani, M.J., Zelna, C.L. and Anderson, J.A. (2004), Assessing Student Learning and Development: A Handbook for Practitioners, National Association of Student Personnel Administrators (NASPA), Washington, DC. Brown, G., Bull, J. and Pendlebury, M. (1997), Assessing Student Learning in Higher Education, Routledge, Oxon and New York, NY. Callison, D. (2000), “Rubrics”, School Library Media Activities Monthly, Vol. 17 No. 2, pp. 4-36, 42.
- 12. Campbell, A. (2005), “Application of ICT and rubrics to the assessment process where professional judgement is involved: the features of an e-marking tool”, Assessment &Evaluation in Higher Education, Vol. 30 No. 5, pp. 529-37. Carr, Nathan T. (2011). Designing and Analyzing Language Tests. Oxford: Oxford University Press. Cambridge ESOL (2011). Using the CEFR: Principles of good practice. Retrieved from http://www.cambridgeenglish.org/images/126011-using-cefr-principles-of-good-practice.pdf Council of Europe (1992). Transparency and coherence in language learning in Europe: Objectives, evaluation, certification. Report on the R¨uschlikon Symposium. Strasbourg: Council of Europe. Fraser, L., Harich, K., Norby, J., Brzovic, K., Rizkallah, T. and Loewy, D. (2005), “Diagnostic and value-added assessment of business writing”, Business Communication Quarterly, Vol. 68 No. 3, pp. 290-305, available at: http://bcq.sagepub.com/cgi/content/abstract/68/3/290(accessed May 2, 2007). Gopinath, C. (2004), “Exploring effects of criteria and multiple graders on case grading”, Journal of Education for Business, Vol. 79 No. 6, pp. 317-22. Little, D. (2011). The common European framework of reference for languages: A research agenda. Language Teaching, 44(3), 381-393. Reiner, C., Bothell, T., Sudweeks, R. & Wood, B. (2002). Preparing effective essay questions. New York: New Forums Press. Shaw, S. & Weir, C. (2008). Examining Writing. Cambridge: Cambridge University Press.