Teaching with Technology Idea Exchange (TTIX) Presentation 06.10.10

Creation of a Statewide Survey of Incentives & Disincentives for the use of OpenCourseWare in Utah Anne Arendt

Research Questions (a) Perceived incentives for use of OCW by the Utah adult population? (b) Perceived disincentives that prevent use of OCW by the Utah adult population? (c) Diffusion attributes that contribute to the adoption (incentives) of OCW in Utah? (d) Diffusion attributes that contribute to rejection (disincentives) of OCW in Utah?

Overall Descriptive study that employed a survey method. Consisted of three stages: a preliminary Delphi technique questionnaire based on Rogers (2003) attributes of innovation, a pilot study, and the primary study.

Delphi Technique Survey instrument developed using attributes that emerged from a Delphi technique Input from experts in the OCW field. Eleven experts where asked to participate and five were actively involved: Marion Jensen, Ahrash Bissell, Terri Bays, Steve Carson, and Andy Lane.

Delphi Initial identify potential incentives and disincentives for end-user OCW use in the form of two questions: (a) in your opinion, what are incentives for potential users of OCW to make personal use of the resource? (b) in your opinion, what are disincentives for potential users of OCW to make personal use of the resource?

Delphi Timeframe The Delphi analysis occurred in three phases: (a) initial question creation, which began on June 12, 2008; (b) compilation and rating, which began on July 3, 2008; and (c) review, which began on July 17, 2008 and ended on July 25, 2008.

Phenomenological Method To compile the results of the experts a phenomenological research method, which describes the lived experiences for several individuals about a concept or phenomenon, was used.

Phenomenological Method (a) list incentives or disincentives, (b) determine the meaning of the statements, (c) identify common characteristics or essentials, and (d) create descriptive statements of essential themes. Throughout this process the experts were asked for input a total of four times.

Delphi Step 2 compiled responses sent to expert rate their agreement with each statement on a five-point Likert scale where 1 represents strongly agree and 5 represents strongly disagree. Statements with a mean of 3.5 or higher and a standard deviation below 1 were designated as areas of agreement.

Delphi Step 3 Areas of agreement were compiled and sent to the group of experts for review with an area allocated where they could identify additional items they believe should have been included that differ from the general tendency but have justification.

Delphi Step 4 Finally, additional items for the survey were again sent out for final review and commentary.

Categorization The results were then distributed into categories of Roger’s innovation attributes: (a) relative advantage, (b) compatibility, (c) complexity, (d) trialability, and (e) observability.

Categorization Although the items were not separated into Roger’s innovation attributes on the end-user distribution survey, these categorizations were used when statistical analysis is performed.

Pilot Test Pilot tested with 44 individuals. Cronbach’s alpha was calculated to assess inter-item consistency for the pilot test and required a reliability of .70 or higher before the survey instrument would be used (Schumacker, 2005).

Cronbach’s Alpha Cronbach’s alpha was calculated to assess inter-item consistency for the N = 44 pilot test and required a reliability of .70 or higher before the survey instrument would be used (Schumacker, 2005). Modifications to the questions were made as necessary based on the results of pretesting.

Cronbach Results Based on the pilot study ( M = 44) categorization by Roger’s attributes and Cronbach’s alpha assessment, two questions were removed. (1) availability of alternative methods to find information online, and (2) the need to purchase books or items not provided online.

Formal Survey Survey sent via postal mail to a randomized group of 753 individuals residing in Utah between the ages of 18 and 64. Names and addresses, along with associated gender, ethnicity, income, age, education, and occupation were obtained from Alesco Data Group, LLC of Fort Myers, Florida.

Sample Size the survey sample size was based on numbers used for inferential statistics. a minimum of 95% confidence level needed to be obtained with a confidence interval, otherwise known as a range of scores or margin of error, of 8%.

Sample Size Based on the Utah population of 1,383,605 for high school graduates ages 18 to 64 in 2006 (U.S. Census Bureau, 2007), a sample size of 150 was necessary to achieve a confidence level of 95% and a confidence interval of 8%. Therefore, 753 surveys were sent out with three follow-up letters.

Sample Size The percentage response rate needed from the 753 surveys was 19.92%. It should also be noted that Alesco data is deemed 90% deliverable, so with that in consideration a percentage response rate of 22.16% would have been required of the deliverable 678 surveys.

Survey Package (a) a cover letter: importance of the participant, incentives offered, purpose, confidentiality, & completion time; (b) a statement of consent, (c) the survey with a unique identification number that tied the survey results back to the demographic variables, and (d) a prepaid addressed envelope for return of the survey (e) a monetary incentive (in this case $.50)

Reminder 1 The first follow up letter was sent via postal mail two weeks after the study introduction. The purpose of this letter was to thank those who have already completed and returned their survey package and remind those who had not yet done so.

Reminders 2 and 3 Second and third follow-up letters were delivered via postal mail to non-respondents on the third and fourth weeks after the study introduction. In the last follow-up letter, instructions were included for requesting another copy of the survey. Three individuals requested new copies of the survey via the email method specified.

Received Back Of 753 surveys set out across Utah, 35 were returned as undeliverable, leaving a total of 718 deliverable surveys. Of the deliverable surveys, 180 responses were received, for an overall response rate of 25.06%. Of the 180 received, 140 were deemed usable.

Received Back Five survey responses were removed at the request of the recipient leaving 175. Ten of the remaining 175 responses were removed due to missing over 20% of the survey answer values, leaving 165 total responses. A category of “do not know” eliminated another 25 responses, leaving 140 total responses.

Confidence With only 140 of the surveys being deemed usable, that number dropped to a confidence level of 95% and a confidence interval of 8.28%.

During Survey A Cronbachs’ alpha was also run at completion of the survey data collection to assess the categorization by Roger’s attributes. A Cronbach’s alpha over .70 was the target. This was achieved for all categories for both incentives and disincentives on all Roger’s attributes.

During Survey Wave Analysis On order to assess the consistency of data throughout the survey period, a wave analysis was done for incentives and disincentives by week for overall means by Roger’s attributes. Although there was some variance in the means and standard deviations, overall the results were consistent.

Results Assessment In addressing the perceived incentives for use of OpenCourseWare by the Utah adult population, statistics were given for overall incentives and frequency for each independent incentive. Descriptive statistics of survey responses for incentives were also supplied. The same was done for disincentives.

Correlations In addressing incentives in the Use of OpenCourseWare in Utah by age, income, gender, education, county, occupation, and ethnicity a variety of correlations were used based on the data type. .

Correlations Pearson product-moment correlations were used between incentives and age and also incentives and income. Point bi-serial correlation coefficients were used between incentives and gender. Spearman’s RHO was used for correlation coefficients between incentives and education. Eta Correlation was used between incentives and county, incentives and occupation, and incentives and ethnicity. The same was done for disincentives.

Roger’s Attributes In considering the diffusion attributes that contribute to the adoption (incentives): Descriptive statistics for incentives as categorized by Roger’s attributes of innovation, overall means for incentives by Roger’s attributes,

Roger’s Attributes frequency for incentives based on Roger’s attributes of innovation, multiple regression analysis on incentives and Roger’s Attributes, and multiple regression analysis of weighted incentive mean and Roger’s attributes. The same was done for disincentives.

Drawback One drawback of this study, however, was that there were not enough results to make use of logistical regression a possibility. As it stands, the multiple regression analysis was not statistically valid; however, the participant numbers were not high enough to constitute logistical regression, although it would have been ideal. Due to this, attempts to quantify the role Roger’s attributes played in predicting user response were not possible.

Open Ended Questions Results of Open-Ended Questions The results of open-Ended questions were compiled in their own section. It was anticipated that results would be categorized but that was not necessary as there were only three open ended comments.

Threats to Validity & Reliability Statistical Validity Threats to external validity involve the extent to which the results can be generalized to other people and places Threats to internal validity have do with selection, where the groups end up not equivalent, and attrition, where some participants do not complete the survey.

Threats to Validity & Reliability History As each participant only completed the survey once there was not a possibility for validity errors between surveys, but there may be events that occur in the 6-week duration of the study itself-which could affect early responders as compared to late responders . Due to this, a wave analysis was done

Threats to Validity & Reliability Sampling Error Concerning sample selection, it may be faulty to assume the similarity of the participants involved. One way to remedy this would be to perform additional surveys of other groups and compare results. For the purposes of this study sampling error is reduced through the use of a large sample.

Threats to Validity & Reliability Response Error According to Dillman (2000), the single most serious limitation to direct mail data collection is the relatively low response rate. Response rates are often only about 5 to 10 percent. Individuals may choose not to respond, which could cause bias in the responses that are received and cause non-response error.

Threats to Validity & Reliability To discourage this from occurring, the survey package included a monetary incentive of 50 cents in the initial survey package because research has shown consistently that inclusion of a small financial incentive can improve response rates significantly over promised incentives, nonmonetary incentives or no incentives (Dillman, 2000; StatPac, 2007).

Threats to Validity & Reliability Although higher monetary rewards may have created even higher response rates, this figure was within the budget allocated for this study. Three follow up letters were also sent to those who had yet to respond, encouraging their participation (Dillman; Miller & Smith, 1983). For this study, a response rate of 25.06% was achieved, well beyond typical response rates noted above.

Threats to Validity & Reliability To address any potential non-response error that still might occur, a comparison of non-respondents and respondents was performed to assess similarities in the demographics in an effort to address any potential discrepancies in the responding sample.

Threats to Validity & Reliability If the non-respondents do not appear different then the results can be generalized to the sample and population (Lindner, Murphy, & Briers, 2001; Lindner & Wingenbach, 2002; Miller & Smith, 1983). There were no differences greater than 10% between respondents and non-respondent demographics.

Threats to Validity & Reliability Response error could also be due to uninformed response error as well. where individuals feel obligated to express an opinion for which they have little or no information was minimized by offering a do not know option. Twenty-five responses were not included in the overall results due to respondents making use of the do not know option.

Threats to Validity & Reliability Reliability internal consistency reliability because the instrument was only administered once, in a singular version, and each participant completed the survey only once. Cronbach’s alpha calculated pilot test and required a reliability of .70 or higher before the survey instrument was used (Schumacker, 2005).

Threats to Validity & Reliability Measurement Error questionnaire had clear, unambiguous questions and response options. Does not completely remove the potential of measurement error, which results when respondents fill out surveys, but do not respond to specific questions.

Threats to Validity & Reliability In order to avoid processing errors the data will be backed up and all calculations, sorts or summaries were run twice to ensure the same results are obtained.

Threats to Validity & Reliability Coverage Error In order to avoid coverage error the sample was obtained from Alesco data, which is deemed 90% deliverable. An updated randomized list was obtained in April 2008 to ensure the timeliness of data.

Issue One challenge with this technique was encouraging and maintaining expert involvement (5 of 11 and some limited) Process of developing the questions took 42 days due to waiting for responses from experts who stated they would submit feedback.

Issue Of the 11 individuals asked to participate in the Delphi technique, five opted to participate: Marion Jensen, Ahrash Bissell, Terri Bays, Steve Carson, and Andy Lane. Andy Lane and Steve Carson participated in all phases

Issue A second limitation was the number of participants in the survey; this occurred in part because the study was self-funded. At the same time the self-funding was a limitation, however, it was also a benefit in that there were no incentives to obtain any particular set of results. Ideally, additional studies can be done with larger sample sets in the future.

The Results Full results of the survey can be found at: ARENDT, A., SHELTON, B.. Incentives and Disincentives for the Use of OpenCourseWare. The International Review of Research in Open and Distance Learning, North America, 10, oct. 2009. Available at: http://www.irrodl.org/index.php/irrodl/article/view/746/1393 .

The Results Printable results: http://www.irrodl.org/index.php/irrodl/rt/printerFriendly/746/1393 The full dissertation for which this research was done can be found at: http://digitalcommons.usu.edu/cgi/viewcontent.cgi?article=1379&context=etd

Themes that Emerged In regard to incentives three themes emerge: (a) self-directed knowledge and learning, (b) convenience, and (c) quality. In regard to disincentives five themes emerge: (a) lack of support, (b) no valid certification, (c) topic issues, (d) lack of content, and (e) lack of resource knowledge.

Outcomes Learning and knowledge are perhaps the most significant incentives for using OpenCourseWare (OCW). However, based on this study, individuals are not driven to use OCW as a precursor to attending a particular institution or to taking a particular traditional class as these questions were asked specifically on the survey

Thoughts Institutions offering OCW could perhaps work to transition some OCW users into degree-granting paid programs by (a) noting available degrees or courses associated with the class the individual is reviewing or (b) permitting a more flexible model of institution entry where individuals could enter into a program at their level of competency.

Thoughts A “test drive” model can be developed to promote or market an institution, using OCW as a maven trap (Gladwell, 2002). Implementing this model would help users to keep their educational costs down, while receiving desired support and valid certification.

Thoughts Offering a flexible entry model into traditional at-a-cost education could be accomplished by offering some type of testing to determine if the OCW user comprehended and mastered the course objectives.

Thoughts If testing is offered, the OCW website could suggest other OCW courses of potential interest as well as provide information about associated degrees or traditional instructor-led courses that seem to be a good fit.

Thoughts The users could find their personal level of competency using measurable assessments. Once the users reached their maximum capability, the results message could explain the potential benefits of traditional instructor-led education for areas they need more help with, noting that although there would now be a cost there would also be support as well as acknowledged and accredited certification or degrees granted.

Thoughts The site could also note traditional at-a-cost classes for which there are no OCW alternatives, yet are practical for their area of interest. This may include classes for which there is extensive lab time, expensive equipment requirements, or requisite instructor-led time.

Thoughts It should be noted that according to this study there is no direct relation between the amount of education a potential OCW user has and the incentives for OCW use, so institutions might also want to re-assess their presumptions relating to prior educational attainment in relation to who may be using, and potentially mastering, OCW materials.

Lack of content or topic issues is another area that surfaced as a disincentive. This disincentive could, in part, be remedied by elevating the status of current OCW/OER recommendation engines such as ccLearn DiscoverEd

Thoughts CW websites should make their content available to these recommendation engines via tags for their content and should reciprocally link to one or more of these sites.

Thoughts Although users might leave one particular institutional site in favor of content in another, they are encouraged to continue their pursuit of knowledge, and this is one of the ultimate goals of OCW and the open educational resources movement.

Thoughts In marketing efforts it is suggested that institutions follow Rogers’s advice for campaign communications. Campaign communications include (a) using formative research to understand the intended audiences and campaign messages, (b) setting specific and realistic campaign goals, (c) using audience segmentation to create more homogenous audience groups, and (d) designing mass media messages that trigger interpersonal network communication to occur.

Thoughts Equally, institutions will want to identify potential opinion leaders, change agents, and champions. As Rogers notes, opinion leaders provide information and advice about innovations to many individuals in the system (2003, p. 27). Change agents influence an individual’s decisions toward the innovation (2003, p. 27). Champions put their weight behind an innovation, thus overcoming indifference or resistance (2003, p. 414).

Thoughts Rogers asserts that mass media is best for communicating at the knowledge acquisition stage to inform potential users of the innovation, and interpersonal communications are best used at the persuasion stage to influence potential users. Institutions will want to consider marketing OCW and other related open educational resources as technology clusters to encourage more rapid diffusion results.

Thoughts This study is meant to spark thinking – not necessarily conclusively answer questions. It is my hope that this presentation and information has sparked ideas and action toward further research in this area.

Teaching with Technology Idea Exchange (TTIX) Presentation 06.10.10

More Related Content

Viewers also liked (9)

Similar to Teaching with Technology Idea Exchange (TTIX) Presentation 06.10.10 (20)

More from Anne Arendt (7)

Recently uploaded (20)

Teaching with Technology Idea Exchange (TTIX) Presentation 06.10.10