Rosetta Stone for Language Learning: Yay or Nay?
Download as PPTX, PDF3 likes2,007 views
- The document summarizes a presentation given by Gillian Lord comparing the effectiveness of Rosetta Stone for learning Spanish as a second language. - Lord collected data from university students learning Spanish through Rosetta Stone, in a classroom, or both, and assessed their linguistic outcomes, attitudes, and perceptions of the materials. - Based on her data and criteria including professional assessments, empirical studies, and student feedback, Lord found both benefits and limitations to the Rosetta Stone approach for language learning.
![My data – Learning Environments
Classroom
• In-tact section
• Followed regular
syllabus with
standard materials
Rosetta Stone
• Self-selected (required by
IRB)
• Not required to attend
any regular class
• Used RS package
(“Conversational
Spanish”):
– 16-week course designed
to cover material
comparable to a face-to-face
beginning class
– 6 units of Rosetta Stone®
Version 4 TOTALe®
Spanish, each has 4
lessons [Level 1, half of
Level 2]
– Minimum of 6 Rosetta
StudioTM sessions
– Minimum of 8 hours in
Rosetta WorldTM
– Monitoring of program
access and time on task
RS + Class
• In-tact section of
Beginning Spanish
class
• Same instructor as
control group
• Used Rosetta Stone
materials as their
textbook
– (including all
features
described for RS
group)](https://image.slidesharecdn.com/actflrsglord-141118120139-conversion-gate01/85/Rosetta-Stone-for-Language-Learning-Yay-or-Nay-7-320.jpg)
![Professional Assessments
• Godwin-Jones (2007, 2009)
– Traditional computer-based training programs often informed by technology specialists,
not SLA specialists
– Rosetta Stone provides visualization feedback for specific sounds (pronunciation); mostly
accurate
• Lafford, Lafford & Sykes (2010)
– Evaluate if programs provide the tools necessary for effective language learning, based on
features that research has shown to be important (interaction, relevant contextualization
of language, etc.)
– “… these products do not incorporate a number of the [necessary] research-based
insights (e.g., the need for culturally authentic, task-based activities) that informed SLA
scholars might have given them.”
• Santos (2011)
– Lack of context
– General inability to respond to spontaneous student speech
– What Rosetta Stone calls interaction is “a rather poor and limited version of what one
would encounter in a real-life conversation”
• DeWaard (2013)
– Based on personal experience, professional reactions
– “Not a viable replacement of current instruction at the postsecondary level”](https://image.slidesharecdn.com/actflrsglord-141118120139-conversion-gate01/85/Rosetta-Stone-for-Language-Learning-Yay-or-Nay-11-320.jpg)
![Student Impressions of Materials
• Comments from English interviews coded for
mentions of the Rosetta Stone materials
– 42/181 comments in RS group
– 80/124 comments in RS+C group
– Difficult to classify as categorically positive or
negative
• Main themes:
1. Technology [Flexibility, Ease of use, Glitches]
2. Approach to language learning](https://image.slidesharecdn.com/actflrsglord-141118120139-conversion-gate01/85/Rosetta-Stone-for-Language-Learning-Yay-or-Nay-17-320.jpg)
![Sample student comments
(Ease of Use)
• Because it’s a lot of visual stuff,
and I feel like I’m a visual learner.
• I like Rosetta Stone a lot. It’s
pretty easy to use.
• World is good, I do the games.
• Oh, and you can do it on your
iPad … so I’d do that a lot, like
take it to people’s houses, and
they’ll all be like watching TV, and
I would be like, doing my Rosetta
Stone, so it’s very versatile.
• I like how it’s like uh, more like a
game, so I’m more willing to
actually do it…
• It was just kind of a lot harder [to
use] than I expected.
• I just didn’t show what words I
needed to use before it.
• I [don’t like] the lack of human
interaction.
• Sometimes it’ll show the person
speaking, and sometimes it’ll say
like he or she, and sometimes it’ll
be I. And I couldn’t tell the
difference.
• …but it is tedious, a lot of it.](https://image.slidesharecdn.com/actflrsglord-141118120139-conversion-gate01/85/Rosetta-Stone-for-Language-Learning-Yay-or-Nay-19-320.jpg)
![Student Impressions
(Language learning in general)
• I feel like it’s more like how you
naturally learn the language instead
of like, “These are … your vocabulary
words this week”.
• I think the Studio session was the
most interesting thing.
• Speaking it for sure [is hardest],
because when you see it on paper it’s
a little bit… but when it gives you
four options, it’s easier to be like, it’s
one of these four.
• The program is really good with like
teaching like vocabulary.
• With like vocabulary, it’s like really
good, and you get by.
• It’s just like the grammar, and how to like,
put it together.
• You can’t ask questions.
• You don’t get any writing, and then all of a
sudden there’s one writing thing.
• You have that whole grammar and
conjugation issue on Rosetta, because
they don’t really explain it.
• You really need to have communication
with a real person.
• I would enjoy getting more grammar
lessons … just to get a foundation of
knowledge, then building up on that.
• But I’m glad to be in the class, too,
because it’s really hard by itself.
• I think it’s helpful to have a teacher to
explain like, why things are.
• Rosetta Stone doesn’t like, I don’t know,
hammer it down for you like they would in
a normal grammar class.](https://image.slidesharecdn.com/actflrsglord-141118120139-conversion-gate01/85/Rosetta-Stone-for-Language-Learning-Yay-or-Nay-21-320.jpg)
Rosetta Stone for Language Learning: Yay or Nay?
- 1. Rosetta Stone for Language Learning: YAY or NAY? Gillian Lord University of Florida glord@ufl.edu
- 2. • How many have heard of Rosetta Stone? • How many have used Rosetta Stone? • How many work for Rosetta Stone?
- 3. Why this topic? • Powerful, omnipresent marketing – Asserts that Rosetta Stone is the only way to learn a language • K-12 and higher education programs feeling threatened “Rosetta Stone … spent $98.5 million on advertising in 2011, up from $70.5 million in 2010, according to Kantar Media” www.nytimes.com/2012/06/20/bus iness/media/rosetta-stone-ads-emphasize- fun-not-efficiency.html
- 4. GOAL OF THIS PRESENTATION: to provide a global understanding, using multiple data sources, of the advantages and disadvantages of the Rosetta Stone program for learning (Spanish as) a second/foreign language
- 5. How to assess such a program? 1. Professional assessments 2. (Previous) Empirical studies 3. Student attitudes (a priori) * 4. Perceived quality of materials * 5. Affective factors * 6. Outcomes – The “Does it work?” question, considering learning in terms of: a) Self-perceived communicative abilities * b) Quantitative measures of communicative abilities * c) Qualitative measures of communicative abilities * * = my data
- 6. My data – Participants • University of Florida students enrolled in Beginning Spanish 1 (avg. age = 20) – L1 English – No other L2 proficiency (beyond h.s. requirement) – Course designed for those with NO prior Spanish instruction • 3 environments: – Classroom (C): N=4 – Rosetta Stone (RS): N=4 – Classroom+Rosetta Stone (RS+C): N=4 Original population had 20-25 participants in each of the three groups.
- 7. My data – Learning Environments Classroom • In-tact section • Followed regular syllabus with standard materials Rosetta Stone • Self-selected (required by IRB) • Not required to attend any regular class • Used RS package (“Conversational Spanish”): – 16-week course designed to cover material comparable to a face-to-face beginning class – 6 units of Rosetta Stone® Version 4 TOTALe® Spanish, each has 4 lessons [Level 1, half of Level 2] – Minimum of 6 Rosetta StudioTM sessions – Minimum of 8 hours in Rosetta WorldTM – Monitoring of program access and time on task RS + Class • In-tact section of Beginning Spanish class • Same instructor as control group • Used Rosetta Stone materials as their textbook – (including all features described for RS group)
- 8. Data used here (1) Quantitative Data • Portion of Spanish CLEP test (grammatical competence) • Versant Proficiency Test in Spanish (oral proficiency) • Attitude survey (Likert, style, pre- and post) • Acoustic analysis of vowel production in Spanish interviews – 10 tokens of /e/ extracted from each participant’s interview. • Tokens separated for male and female • Each formant measured at midpoint using Praat • F1 and F2 extracted using a script and compared to standard native values
- 9. Data used here (2) Qualitative Data • Fluency analysis of Spanish interviews • Total number of words spoken • Number of Spanish words; Number of English words • Number of dysfluencies • Lexical density (number of unique Spanish words) • Number of fillers/non-lexical items • Content analysis of English interviews • Affective factors • Reactions to instructional materials • Student perceived learning • Language communication and use • Other specific language problem or comment
- 10. Criterion 1: PROFESSIONAL ASSESSMENTS OF THE ROSETTA STONE PROGRAM
- 11. Professional Assessments • Godwin-Jones (2007, 2009) – Traditional computer-based training programs often informed by technology specialists, not SLA specialists – Rosetta Stone provides visualization feedback for specific sounds (pronunciation); mostly accurate • Lafford, Lafford & Sykes (2010) – Evaluate if programs provide the tools necessary for effective language learning, based on features that research has shown to be important (interaction, relevant contextualization of language, etc.) – “… these products do not incorporate a number of the [necessary] research-based insights (e.g., the need for culturally authentic, task-based activities) that informed SLA scholars might have given them.” • Santos (2011) – Lack of context – General inability to respond to spontaneous student speech – What Rosetta Stone calls interaction is “a rather poor and limited version of what one would encounter in a real-life conversation” • DeWaard (2013) – Based on personal experience, professional reactions – “Not a viable replacement of current instruction at the postsecondary level”
- 12. Criterion 2: EMPIRICAL STUDIES INTO THE OUTCOMES OF THE ROSETTA STONE PROGRAM
- 13. Empirical studies • Vesselinov (2009) – Commissioned by Rosetta Stone; RS beginning users demonstrated increased knowledge of the language after 55 hours of use. • Nielson (2011) – Self-study programs in workplace; some success among committed users, but extreme attrition. • Stevenson & Liu (2010) – Use of social interaction in web-based language learning tools; lack of ability to engage learners in true interaction; users do not take advantage of network opportunities.
- 14. Criterion 3: STUDENT ATTITUDES TOWARDS THE ROSETTA STONE PROGRAM
- 15. Background questionnaire • Why did you volunteer for the Rosetta Stone class-replacement option? – I heard a lot of good things about RS and wanted to try it. – Can better manage my time and schedule and move more at my own pace without dealing with class. – Sounded more beneficial. – I was going to use my own to supplement education anyway.
- 16. Criterion 4: PERCEIVED QUALITY OF THE ROSETTA STONE MATERIALS
- 17. Student Impressions of Materials • Comments from English interviews coded for mentions of the Rosetta Stone materials – 42/181 comments in RS group – 80/124 comments in RS+C group – Difficult to classify as categorically positive or negative • Main themes: 1. Technology [Flexibility, Ease of use, Glitches] 2. Approach to language learning
- 18. Sample student comments (Flexibility) • Like you’re able to kind of do it like on your own time, you know, I’m not like restricted. • It’s nice not to go to class. • I have always preferred to learn language, like, on my own. • It’s more flexible with my schedule.
- 19. Sample student comments (Ease of Use) • Because it’s a lot of visual stuff, and I feel like I’m a visual learner. • I like Rosetta Stone a lot. It’s pretty easy to use. • World is good, I do the games. • Oh, and you can do it on your iPad … so I’d do that a lot, like take it to people’s houses, and they’ll all be like watching TV, and I would be like, doing my Rosetta Stone, so it’s very versatile. • I like how it’s like uh, more like a game, so I’m more willing to actually do it… • It was just kind of a lot harder [to use] than I expected. • I just didn’t show what words I needed to use before it. • I [don’t like] the lack of human interaction. • Sometimes it’ll show the person speaking, and sometimes it’ll say like he or she, and sometimes it’ll be I. And I couldn’t tell the difference. • …but it is tedious, a lot of it.
- 20. Sample student comments (Tech glitches) • I’m still having problems with the computer. • I can’t get the microphone to work… • I was doing my Studio session and… I had no audio, like, I could hear them, but they couldn’t hear me the whole time.
- 21. Student Impressions (Language learning in general) • I feel like it’s more like how you naturally learn the language instead of like, “These are … your vocabulary words this week”. • I think the Studio session was the most interesting thing. • Speaking it for sure [is hardest], because when you see it on paper it’s a little bit… but when it gives you four options, it’s easier to be like, it’s one of these four. • The program is really good with like teaching like vocabulary. • With like vocabulary, it’s like really good, and you get by. • It’s just like the grammar, and how to like, put it together. • You can’t ask questions. • You don’t get any writing, and then all of a sudden there’s one writing thing. • You have that whole grammar and conjugation issue on Rosetta, because they don’t really explain it. • You really need to have communication with a real person. • I would enjoy getting more grammar lessons … just to get a foundation of knowledge, then building up on that. • But I’m glad to be in the class, too, because it’s really hard by itself. • I think it’s helpful to have a teacher to explain like, why things are. • Rosetta Stone doesn’t like, I don’t know, hammer it down for you like they would in a normal grammar class.
- 22. Criterion 5: AFFECTIVE FACTORS AND EMOTIONAL REACTIONS TO THE ROSETTA STONE PROGRAM
- 23. Affective factors • Comments from English interviews coded for any mention of affect/emotion – Related to using the materials, learning, etc. and how the learners felt in that respect • 33/181 comments in RS group – 8 positive towards RS, 8 negative, 17 unrelated • 24/123 comments in RS+C group – 2 positive towards RS, 9 negative, 13 unrelated
- 24. Student reactions/emotions • I don’t have a problem with staying focused on my computer at all. • I like not having a textbook to worry about. • I like it a lot. • I’m kind of struggling. • I didn’t like the stories. • I’m always just frustrated because I’m like, I don’t really understand it. • I feel like it should be more structured.
- 25. Criterion 6: LINGUISTIC OUTCOMES OF THE ROSETTA STONE PROGRAM a) Self-perceived communicative abilities
- 26. Perceived learning • Comments from English interviews coded for any mention of perceived learning or outcomes • 34/181 comments in RS group – 9 positive assessments, 22 negative, 3 unrelated to materials • 14/123 comments in RS+C group – 8 positive assessments, 3 negative, 3 unrelated
- 27. Perceived learning • I understood like the vocab. • I feel like I’m slowly learning and progressing to learn the language. • I do like the studio sessions… When I do do them, like I feel like it helps me more. • I can understand almost always what is being said. • Learning a language just like on your computer as opposed to like in person can actually be effective. • Rosetta Stone doesn’t give you too much, like, actual instruction so you don’t learn. • I don’t know how to use words in Spanish. • I’m just kind of like nervous about going back to class. • I don’t feel as confident as I did at that last meeting, honestly. • We didn’t ever have to like memorize the conjugations, so I don’t know them, so I can’t say that it was super effective. • It was just kind of a lot harder than I expected.
- 28. Perceived learning (pre-post changes on relevant items from attitude survey) – Significant decrease among RS on item #11: • “Interacting via chat or telephone is comparable to interacting face-to-face.” – Significant increase among RS and RS+C on item #19: • “I would prefer to learn a language on my own time and at my own pace than in a group or classroom setting.”
- 29. Criterion 6: LINGUISTIC OUTCOMES OF THE ROSETTA STONE PROGRAM b) Empirical measurements of communicative abilities
- 30. CLEP test Average scores (converted to %) p = 0.165 38.65 39.17 47.50 100.00 90.00 80.00 70.00 60.00 50.00 40.00 30.00 20.00 10.00 0.00 Classroom Rosetta Stone RS+class
- 31. Versant test Average scores (converted to %) p = 0.620 27.08 26.25 20.00 100.00 90.00 80.00 70.00 60.00 50.00 40.00 30.00 20.00 10.00 0.00 Classroom Rosetta Stone RS + Class
- 32. Pronunciation Acoustic analysis of vowel /e/ • No significant difference between groups or test times. – Control males shows slight trend toward more native-like F1 values in final interview. RS = RS+Class = Classroom?
- 33. Criterion 6: LINGUISTIC OUTCOMES OF THE ROSETTA STONE PROGRAM c) Discourse analysis reflecting communicative abilities
- 34. Final interview - Classroom INTERVIEWER: Cuéntame, ¿qué te gusta hacer en tu tiempo libre, o los fines de semana? SL: Repitas, please. INTERVIEWER: ¿Qué te gusta hacer? SL: Qué te gusta hacer… INTERVIEWER: ¿Te gusta ir a películas? ¿Te gusta escuchar música? SL: Uh, ¿fin de semana? INTERVIEWER: Sí. SL: Uh, sí. En fin de semana, yo… yo estudio, uh, mucho. INTERVIEWER: ¿Sí? SL: Uh, para mis exámenes. Sí. Yo tengo muchos examines en química orgánica, biología, y laboratorio. Uh, sí. Mucho, uh… no, muy ocupado. So, no películas, no, uh, deportes. INTERVIEWER: ¿Cuál fue la última película que viste? SL: Cuál te… INTERVIEWER: La última vez, the last time, que viste una película. SL: Phew… Hmm. Let’s see… dos menses. INTERVIEWER: Meses, mhm. SL: Meses. Ago. ¿Cómo se dice “ago”? INTERVIEWER: Hace. Hace dos meses. SL: Hace, sí. INTERVIEWER: Wow. SL: Yo no… yo no veo muchas películas en Gainesville. INTERVIEWER: ¿Qué película fue esa, hace dos meses? ¿Cómo se llamaba? SL: Uh, el pelí—la película… ¿cómo se dice “was”? INTERVIEWER: Era, o fue. SL: Era. La película era… INTERVIEWER: ¿No te acuerdas? SL: Yo no… sí.
- 35. Final interview – Rosetta Stone INTERVIEWER: Mhm, ¿y qué haces en Gainesville? SH: Um… you’re going to have to forgive me, my mind’s like blown… Um, yo estoy estudiar. INTERVIEWER: ¿Tú estudias? ¿Y qué más? SH: Yo trabajo en un restaurante de Dragonfly. INTERVIEWER: Y, ¿con mucha frecuencia, vas de compras? SH: Yo no entendí, repetirlo, por favor. INTERVIEWER: ¿Con mucha frecuencia, vas de compras? “Ir de compras” significa go shopping. SH: All right, say that one more time, please. INTERVIEWER: ¿Con mucha frecuencia, vas de compras? SH: Uh, no, uh, no voy a… what did you, how did you say “to go shopping”? INTERVIEWER: Ir de compras. SH: No voy de compras. INTERVIEWER: Y, ¿qué vas a hacer este verano? SH: Este verano, yo voy a visitar Brazil. INTERVIEWER: Vas a visitar Brazil, y ¿vas a estudiar en Brazil? SH: No, um, yo voy a trabajar en Brazil. INTERVIEWER: Y, em, ¿qué más vas a hacer en Brazil? ¿Vas a leer, vas a jugar deportes? SH: What am I going to do in Brazil? I thought I just answered that. INTERVIEWER: ¿Solo trabajar? SH: I don’t know, I’m going on a missions trip, I don’t know how to express that in Spanish, but… INTERVIEWER: Pues, buena suerte, muchas gracias.
- 36. Ratio of L1/L2 words 0.26 0.83 0.68 1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00 Ratio of English-to-Spanish words used, by group Control Average RS + class Average Rosetta Stone Average 0 = no English words produced 1 = 1 English word produced for every Spanish word
- 37. Assistance requests Average # of clarification / assistance requests by group 2.06 1.03 0.11 1.75 0.61 3.24 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 # Clarification requests in Spanish # Clarification requests in English Control Average RS + class Average Rosetta Stone Average
- 38. SUMMARY
- 39. Yay or Nay? 1. Professional assessments 2. Empirical studies 3. Student attitudes (a priori) 4. Perceived quality of materials 5. Affective factors 6. Outcomes a) Self-perceived communicative abilities b) Empirical measurements of communicative abilities c) Discourse analysis reflecting communicative abilities
- 40. What about overall time on task? GROUP Completion Rate Average Score Total Course Usage (hours) Total Class Time (hours) Classroom 96.99% 90.77% 70.00 39.00 RS+C 93.67% 98.63% 32.81 37.25 RS 97.67% 95.88% 30.69 NA
- 41. But… ?
- 42. IN CONCLUSION…
- 43. 1. More research is needed. • Further professional assessments of the programs, as new features are added • Continuing analysis of current data • Empirical studies of outcomes and effectiveness – Attitudes and reactions – Linguistic outcomes – Larger, more diverse populations
- 44. 2. Rosetta Stone can be effective. • In certain circumstances: – Introduction and exposure to new languages – Refresh skills previously acquired – Excellent vocabulary presentation and practice – Flexible for varied populations – Highly motivated/diligent learners more apt to make progress Undoubtedly better than nothing!
- 45. 3. BUT… • Based on multiple data sources, Rosetta Stone does not appear to be, or have the potential to be more effective, faster, easier or (necessarily) more enjoyable than other methods, such as common classroom-based approaches.
- 46. Keep an open mind about RS and other similar products. – CALL continues to expand and develop. – Rosetta Stone has acquired new tools and capabilities. – Knowledge of another language is always valuable. But don’t believe everything the ads promise. – Claims are largely unfounded so far. – Immersion or classroom experiences remain most promising methods of acquiring FL proficiency.
- 47. Thank you. glord@ufl.edu Special thanks to: • UF College of Liberal Arts & Sciences • UF CLAS Humanities Scholarship Enhancement Fund • Carlos Enrique Ibarra (statistics) • Caroline Reist, Brandon Shufelt, Keegan Storrs, Diana Wade (RAs) • Laura Bradley, Lisa Frumkes (Rosetta Stone)
- 48. Works Cited • DeWaard, L. (2013). “Is Rosetta Stone a viable option for L2 learning?” Forthcoming in ADFL Bulletin. • Godwin-Jones, R. (2007). “Emerging technologies; Tools and trends in self-paced language instruction. Language Learning and Technology,” 11(2), 10-17. Retrieved 26 September 2012 from http://llt.msu.edu/vol11num2/emerging/ • Godwin-Jones, R. (2009). “Emerging technologies: Speech tools and technologies. Language Learning and Technology,” 13(3), 4-11. Retrieved 26 September 2012 from http://llt.msu.edu/vol13num3/emerging.pdf • Lafford, B., Lafford, P. & Sykes, J. (2007). “Entre dicho y hecho …: An assessment of the application of research from second language acquisition and related fields to the creation of Spanish CALL materials for lexical acquisition.” CALICO Journal, 24(3), 427-529. • Nielson, K. B. (2011). “Self-study with language learning software in the workplace.” Language Learning and Technology, 15(3), 110-129. Retrieved 26 September 2012 from http://llt.msu.edu/issues/october2011/nielson.pdf • Santos, V. (2011). “Review of Rosetta Stone Portuguese (Brazil) levels 1, 2, & 3.”CALICO Journal, 29(1), 177-194. • Stevenson, M. P. & Liu, M. (2010). “Learning a language with web 2.0: Exploring the use of social networking features of foreign language learning websites.” CALICO Journal, 27(2), 233-259 • Vesselinov, Roumen. Measuring the Effectiveness of Rosetta Stone. http://resources.rosettastone.com/CDN/us/pdfs/Measuring_the_Effectiveness_RS-5.pdf.
- 50. Sample CLEP test items
- 52. Versant proficiency test • Instrument has been tested for reliability and validity
- 54. Rosetta Stone interface (vocabulary)
- 55. Rosetta Stone interface (grammar)
- 56. Rosetta Stone interface (pronunciation)
- 57. Rosetta Stone interface (World – “play”)
- 58. Rosetta Stone interface (World – “talk”)
- 59. Rosetta Stone interface (World – “explore”)
- 60. Rosetta Stone interface (Studio)
- 61. Attitude survey
Editor's Notes
- #2: https://www.slideshare.net/secret/9YEeyZsMjQbaOg
- #3: How many of you are familiar with Rosetta Stone’s programs? Based on what? How many of you as language teachers have been asked “Does RS work?” How many of you have used Rosetta Stone? Which languages? How long? How effective? How many of you work for Rosetta Stone?
- #4: Huge marketing endeavor According to RS’s 2011 10-K SEC filing nearly 80% of Americans were already familiar with the company/product Asserts that Rosetta stone is superior to any and all other methods of instruction Administrators, even some teachers, are tempted – need to cut budgets, save time, save resources, etc. Stories of school districts replacing language teachers with this program So we need to know if it works! When I have spoken with folks at RS they say they never intended to replace us (and I believe those specific people), but the ads don’t lie. Lest you don’t believe me …
- #5: As you’ll see on the next slides, some of what I’m presenting here is previous research, some of it is my own data from a study I carried out using RS. So I need to make it really clear up front that that study was conducted with Rosetta Stone’s knowledge but they were not involved in the design, data collection or analysis. The Rosetta Stone licenses were purchased at regular price for the quantity I purchased with grant money provided by my institution. But I did work with a few different contacts at RS when purchasing the licenses and setting up the classes and the admin side of it, and I am grateful for their administrative and technical help in setting up the licenses and reading the data reports, learning the system. That said, I should also point out that I have in writing that I am free to share any and all results I obtained, because we all agreed going in that I was approaching this as an academic, pursuing a valid empirical question –what works? What doesn’t _ with no ulterior motives.
- #6: *explain that I’m not using user reviews - not necessarily reliable, don’t know anything about the users, run the gamut. WILL NOT USE MARKETING MATERIALS. Professional assessments of Rosetta Stone Empirical studies investigating Rosetta Stone outcomes Preconceived student attitudes towards Rosetta Stone Perceived linguistic/learning value of materials during and after use of of Rosetta Stone Affective factors/emotional reactions towards Rosetta Stone Outcomes of Rosetta Stone (with this population) The “Does it work?” question, looking at improvement in terms of: Self-perceived communicative abilities Empirical measurements of communicative abilities Discourse analysis reflecting communicative abilities NOTE: RS’s FAQ page lists the “does it work” question, and their answer talks about how millions of learners have “discovered” a language, but doesn’t talk about outcomes. So we need to operationalize “does it work” a bit better.
- #7: Professional assessments of Rosetta Stone Empirical studies investigating Rosetta Stone outcomes Preconceived student attitudes towards Rosetta Stone Perceived linguistic/learning value of materials during and after use of of Rosetta Stone Affective factors/emotional reactions towards Rosetta Stone Outcomes of Rosetta Stone (with this population) The “Does it work?” question, looking at improvement in terms of: Self-perceived communicative abilities Empirical measurements of communicative abilities Discourse analysis reflecting communicative abilities
- #8: Talk about data collection English and Spanish interviews Writings clep
- #12: Evaluate if programs provide the tools necessary for effective language learning, based on features that research has shown to be important (interaction, relevant contextualization of language, etc.)
- #14: Few empirical studies but overall lack of evidence to support the notion that such programs can replace the classroom or can offer anything that a classroom program cannot. VESSELINOV: Rosetta Stone commissioned after 55 hours of Rosetta Stone Spanish, students will “significantly improve” their language skills participants were self-selected study population was older than traditional student age (the average age was 41) . highly educated WebCAPE test, computer Oral Proficiency Interview (OPI) test. The scored an average score of 238 on the WebCAPE which, in most universities, represents language skills less than those of first-semester courses. NIELSON: Unfortunately, data on linguistic outcomes in Nielson’s study are scant, due to her most striking finding: the completion rate. Of the 150 participants involved in using Rosetta Stone, only one (an Arabic learner) completed the third and fourth assessments and the OPI; only six had completed the second assessment (after 100 hours). In fact, of the 150 learners who agreed to participate in the Rosetta Stone portion of the study, only 120 opened their accounts, and only 73 of those ever accessed their accounts. All but 32 stopped using (or never used) the program prior to completing their first ten hours of use. Will tell about the study design for the present study and briefly review previous findings (presented at calico last year, currently under review) before adding additional data
- #16: Result of powerful marketing. We know that our students don’t always have an understanding of what it takes to learn a language! But their good will and enthusiasm toward the product should be noted.
- #21: But to be fair, all LMS/CMS etc. systems have glitches, so this is nothing new or special. Microphone issues were consistent though, even using the USB headsets provided.
- #22: But is choosing from four options really learning a language??
- #24: RS group comments focused on ease/flexibility and how much they liked that, so confounded variables 10 positive out of 304 = 3.3% 17 negative out of 304 = 5.6%
- #27: Unrelated = things like “I like French better, so I feel like I can’t learn Spanish” Interesting trend – most of negative comments about RS emerged after the midway point of the semester, especially as they start realizing they had to go back to a ‘regular’ class Most positive comments about vocabulary IN RS+C positive comments about how they complement each other 17/304 positive = 5.6% 25/304 negative = 8.2%
- #30: Have three empirical measures Clep Versant Acoustic vowel analysis
- #31: Kruskal Wallis Chi-Square= 3.604 p = 0.165 With a mean rank of 4.88 for the Classroom Group, a mean rank of 5.38 for the Rosetta Stone group, and a mean rank of 9.25 for the RS+Classroom group.
- #32: Kruskal Wallis Chi-Square = 0.971 p = 0. 615
- #35: Some comprehension trouble Lots of confirmation checks Tries in Spanish, asks for missing words (exclamation or two in English)
- #36: Able to answer, some need for repetition and clarification Resorts to English as well
- #37: Fluency analysis All interviews 3 groups x 4 participants = 12 participants x 3 interviews = 36 Transcribed and analyzed for fluency measures “Fluency” ends up looking at Number of Spanish words; Number of English wordsl Lexical density (number of unique Spanish words) one striking aspect of the interviews is the quantity of English words used by the different groups. The ratio of English-to-Spanish was calculated by dividing the number of English words by the number of Spanish words; a ratio of 0 would indicate that the entire production was exclusively in Spanish, while a ratio of 1 would mean one English word was produced for every Spanish word. (A ratio greater than 1, which did not occur here, would indicate more English than Spanish used.) The Rosetta Stone + Class group produced approximately eight English words to for every ten Spanish words they produced, while the Rosetta Stone group produced almost seven English words for every Spanish word; in other words, they produce almost as much English as Spanish in their interview responses. The Control Group on the other hand, used less English, with a rate of .26, indicating that they produced only two to three English words for every Spanish word.
- #38: the Control/Classroom group produced twice as many of these requests in Spanish as in English, while the Rosetta Stone and Rosetta Stone + Class groups produced English requests more than twice as frequently as Spanish requests. Taken together, these measures – and particularly those relating to use of English – confirm the impressions of those carrying out the interviews: although all learners were clearly novices who struggled to communicate fluently, the Rosetta Stone group seemed to struggle more and frequently resorted to English, while those who attended class and thus engaged in true communication with instructor and peers, were better equipped to request assistance when needed or attempt to convey their message even in spite of linguistic lacunae. The Rosetta Stone + Class group seems to represent an odd mixture, which in some ways outperformed the Rosetta Stone (only) group, and in others appear to be less proficient than both other groups. Anecdotally, the interviewers found that the interviews with the Rosetta Stone group frequently devolved into simple vocabulary questions (¿Qué es esto? “What is this?”) while the Control group was able to engage, albeit haltingly, in basic conversations; again, the Rosetta Stone + Class group represented an interesting middle ground, with the same limited conversational tools as the Rosetta Stone group but slightly more disposition to form discourse length utterances.
- #40: Professional assessments of Rosetta Stone – unanimously agree that can’t work Empirical studies investigating Rosetta Stone outcomes – very few, and methodologically unreliable, but outcomes doubful Preconceived student attitudes towards Rosetta Stone – very positive, very excited Perceived linguistic/learning value of materials during and after use of of Rosetta Stone – TIE – no way to tell, and no overall preference. Liked flexibility, liked digital, missed human connection and had tech glitches. Affective factors/emotional reactions towards Rosetta Stone – overall positive, with some negative Outcomes of Rosetta Stone (with this population) Self-perceived communicative abilities – overall negative, felt they didn’t learn as much as they had hoped or expected (17/304 positive = 5.6%, 5/304 negative = 8.2%) Empirical measurements of communicative abilities – inconclusive Discourse analysis reflecting communicative abilities – overall negative, much greater problems communicating from RS gropu
- #41: Also, time on task may, in fact, be partly responsible for some of these differences. . Table provides usage data for all participants, including the percentage of assigned material completed, the average score on those materials, the total number of hours logged in the online system (either Rosetta Stone or MySpanishLab, the course management system accompanying the textbook), and for the Rosetta Stone groups, the number of hours they spent in Rosetta World and Rosetta Studio sessions. Also, after considering absences from class sessions, the number of hours spent in class was recorded for the two in-class groups. The Control group averaged 109 hours of exposure over the course of the semester, including classroom hours and homework hours online, while the Rosetta Stone group averaged only 48 hours over the semester. This is not to say that the main factor in language learning is seat time, but it is a factor that must be considered; the Control group had twice as much time to learn, practice and use the language than the Rosetta Stone groups, and that clearly had some impact on the outcomes. Simple hours of exposure, though, would not account for the qualitative differences observed in the oral performance of the groups. http://www.tubechop.com/watch/4051426 <object width="425" height="344"><param name="movie" value="http://swf.tubechop.com/tubechop.swf?vurl=nMAOSZWnLQs&start=5.06&end=11.81&cid=4051426"></param><embed src="http://swf.tubechop.com/tubechop.swf?vurl=nMAOSZWnLQs&start=5.06&end=11.81&cid=4051426" type="application/x-shockwave-flash" allowfullscreen="true" width="425" height="344"></embed></object>
- #44: * But challenges to new studies – fear of reprisal? – NM Dept of Health study
- #45: But flexibility is often outranked by motivation, which we all know (nordic track) can vary
- #46: Skeptical but not cynical So where does that leave us?
- #53: Using 575 native Spanish speakers (from various countries) and 564 Spanish learners, who all took the test; human raters (ACTFL OPI; government-certified SPT; Common European Framework; ILR ) compared ratings across samples to ensure that native speakers score high, and learners score across a range; and to ensure that test results will be consistent for same test-taker if no change in proficiency