Assessing Professional Skill Development In Capstone Design Courses
0 likes4 views
The document summarizes the development and testing of assessments to measure students' professional skill development in capstone design courses. The Transferable Integrated Design Engineering Education (TIDEE) consortium developed four assessments focused on technical, interpersonal, and individual attributes. Initial testing was conducted at the University of Idaho, with 15 students completing a summative assessment. Results showed growth in skills like leadership and adapting to change. Future work will analyze longitudinal data to validate the assessments and measure student perceptions.
Assessing Professional Skill Development In Capstone Design Courses
- 1. Assessing Professional Skill Development in Capstone Design Courses Denny Davis1 , Michael Trevisan1 , Howard Davis1 , Robert Gerlick1 , Jay McCormack2 , Steven Beyerlein2 , Phillip Thompson3 , Susannah Howe4 , Paul Leiffer5 , Patricia Brackin6 , Javed Khan7 1 Washington State University 2 University of Idaho 3 Seattle University 4 Smith College 5 LeTourneau University 6 Rose-Hulman Institute of Technology 7 Tuskegee University Professional skills and behaviors desired by employers in engineering graduates are often difficult to assess in project products. Professional skills highlighted by ABET are professional and ethical responsibility and ability to engage in lifelong learning, but also include a variety of other characteristics that arise through interactions with project stakeholders. The Transferable Integrated Design Engineering Education (TIDEE) consortium has identified a set desired professional attributes and developed a set of professional development assessments for measuring student growth in these areas, providing student-centered feedback, and supporting accreditation efforts. Testing has begun with these instruments and early results are presented. Corresponding Author: Denny Davis, davis@wsu.edu Introduction Capstone engineering design courses are often the setting for the development and refinement of many of the complex, nontechnical skills required to be a high performing engineer. These skills are also reflected in program accreditation standards. For example, ABET requires engineering programs to demonstrate that their graduates understand professional and ethical responsibility and recognize the need for and demonstrate the ability to engage in lifelong learning. However, because professional development skills are complex and multifaceted, they are challenging to assess. The Transferable Integrated Design Engineering Education (TIDEE) consortium of institutions has addressed this challenge by developing assessments that can be used to provide feedback to students about their performance. These same assessments can be aggregated for purposes of program assessment. This work has helped engineering educators bridge the gap between educational theory and classroom practice in capstone engineering design courses. TIDEE’s overall focus is developing and assessing students' capabilities in engineering design, including professional development. To this end, project faculty developed and piloted an integrated set of assessments for teamwork, professional development, design processes, and solution assets1,2 . To date, both teamwork and professional development assessments have been piloted in programs represented by the authors of this paper. Results from implementing and testing the teamwork assessments have been previously discussed3 . Four assessments are also being tested which examine professional development in technical, interpersonal, and individual attributes important to personal and project needs, workplace behaviors, and the ways of being of a reflective practitioner. A key feature of the TIDEE assessments is a web-based implementation that allows instructors to use the assessments in a manner that supports student reflection, is sustainable, and minimizes non-value-added activity. This paper provides a brief description of the four TIDEE professional development assessments, with more detailed description and results from implementation of the summative assessment instrument (Professional Development Achieved) in one capstone program. Background on professional skill assessment is reviewed, results are reported from analyses of student work on the Professional Development Achieved assessment, and future work is outlined.
- 2. Literature Review Professional development skills include a breadth of topics deemed relevant by industry and academic sources. Davis, et al.4 surveyed fifty people from various disciplines (BioEnvE, CE, ChE, ECE, Engr, GeoE, ME, and PetrE) and backgrounds (academic and industry) to provide guidance on the preferred skills of new engineering employees. Topics in the areas of teamwork, ethical behavior, and communication were deemed essential in addition to technical competence. Topics addressed in the TIDEE assessments are derived from the areas of interest expressed in this survey and industry reports5 . These include a desire for lifelong learning, which aligns with ABET criterion (i) and is also embedded in the TIDEE assessments. Table 1 gives twelve professional development abilities/attributes which anchor TIDEE’s professional development assessments. Table 1 divides these into three major areas—technical, interpersonal, and personal— and includes definitions of each ability/attribute. TIDEE Professional Skills Assessments The TIDEE professional skills assessments consist of three formative assessments (Professional Development Planning, Professional Development Progress, and Professional Practices) and one summative assessment (professional development achieved). Professional Development Planning Students begin the Professional Development Planning assessment activity by rating the importance of twelve professional abilities and attributes (Table 1) according to their own perceptions. Subsequently, students provide a rating of their own abilities in each of the twelve areas. Finally, students select an attribute that is important and needs to be developed further to enhance their project success. Students then describe in a few paragraphs their shortcoming(s) in the specified ability/attribute and their plan to overcome the shortcoming. Students are then scored on their understanding of the impact of their identified shortcoming, the quality of plan for overcoming the shortcoming, and the evidence that will tell them that they achieved growth. Professional Development Progress In the Professional Development Progress assessment activity, students provide an update on their progress in achieving the plan identified in the Professional Development Planning assessment activity. In a few short paragraphs, students describe their progress and revised plans for achieving their targeted professional development by specifying details that illustrate the steps taken to achieve their targeted professional development, evidence of impacts of their professional development to-date on the project or team success, and additional steps that will be taken to achieve targeted professional development. Students are scored on their progress to date, the quality of their evidence of progress, and the quality of their newly planned steps. Table 1. Twelve professional development abilities/attributes and accompanying description. Professional Development Ability/Attribute and Description Analyzing information: Applying methods/tools of analysis to understand and predict conditions Solving problems: Formulating, selecting, and implementing actions for optimal outcomes Designing products: Producing creative, practical products that bring value to varied stakeholders Technical Researching questions: Investigating, processing and interpreting information to answer important questions Communicating: Receiving, processing, sharing information in many forms to achieve desired impact Collaborating: Working with a team to achieve collective and individual goals Relating inclusively: Valuing and sustaining a supportive environment for all knowledge and perspectives Interpersonal Leading others: Developing shared vision & plans; empowering to achieve individual & collective goals Practicing self-growth: Planning, self-assessing, and achieving goals for personal development Being a high achiever: Delivering consistently high quality work and results on time Adapting to change: Being aware and responding proactively to social, global, and technological change Individual Serving professionally: Serving with integrity, responsibility and sensitivity to individual and societal norms Professional Practices Students first identify the importance of seven areas of professional responsibility (work competence; financial responsibility; honest communication; health, safety, and well-being; property ownership; sustainability; social responsibility) to their projects. Students then identify their current level of performance in each area of professional responsibility in the context of their project. Finally, students describe important areas of their project in which they have successfully demonstrated professional responsibility and areas that must be addressed. Students are scored on the evidence
- 3. provided for their strong performance and their understanding of opportunity and plan to achieve higher performance in an area of professional responsibility. Professional Development Achieved The Professional Development Achieved assessment is the summative assessment that follows the Professional Development Planning and Professional Development Progress assessments. Students are again presented with the twelve professional development abilities/attributes and are asked to rate their change in perceived importance of each ability and their change in perceived performance in each ability. The rating scale consists of three levels, decreased, no change, and increased. Students then identify the ability in which they experienced the most significant personal growth and describe how they have grown in this ability, how that growth has proven valuable to the project, and how the growth experienced has prepared them for future professional development. Student‘s written reflections are scored using the rubric shown in Figure 1. Students can also receive written comments and suggestions for improvement from the instructor. Figure 1. Scoring Rubric for Professional Development Achieved. Implementation Testing of the professional skills assessment instruments is ongoing at various institutions and will be studied for instrument validation and to guide improvement. The initial data comes from the interdisciplinary engineering capstone design course at the University of Idaho where the instruments were used over the summer and fall semesters of 2009. Students completed the Professional Practices and Professional Development Achieved assignments through the web-based interface and the Professional Development Planning in logbooks. Results from the Professional Development Achieved assessment are reported in the next section. Results Fifteen participants completed the Professional Development Achieved assessment activity. The student ratings of their perceived change in importance and perceived growth in the twelve specified areas of professional development are shown in Table 2. Students indicated if their perception of importance or growth decreased (D), did not change (NC), or increased (I). The frequency of student responses to each category is shown. Note that one student failed to rate perceived growth in the area of analyzing information, so there are only fourteen responses in this category. In the last column, Table 2 contains the number of students that marked each professional skill as the area of most significant personal growth. Students then described how they grew in this ability, how this growth benefitted their project, and how the growth experience will prove valuable in the future. Table 2. Student rating of perceived change in importance and perceived growth in professional development attributes as well as area of most significant growth. Perceived change in importance Perceived growth Ability/ Attribute D NC I D NC I Most significant growth Analyzing information 0 6 9 0 8 6 1 Solving problems 0 7 8 0 6 9 2 Designing products 0 6 9 0 4 11 2 Researching questions 1 8 6 2 6 7 0 Communicating 0 5 10 0 11 4 0 Collaborating 0 6 9 4 4 7 1 Relating inclusively 1 11 3 1 9 4 0 Leading others 1 3 11 2 4 9 4 Practicing self- growth 1 3 11 2 5 8 0 Being a high achiever 0 8 7 0 6 9 1 Adapting to change 0 8 7 0 12 3 3 Serving professionally 0 9 6 0 8 7 1 More data gathering, comparison to growth plans, and analysis of the data is clearly needed, but it is interesting to note attributes that figured prominently in student assessments thus far. Leading others was an attribute that was highly rated in both student perceived increase in importance and increase in growth as well as an area of most personal growth. Since scheduling and performance review are often roles filled by course instructors, it is also not surprising that an open-ended,
- 4. industry sponsored, student led design project would demand more importance be placed on leadership and offer more opportunities for growth in leadership areas. Collaborating was most frequently cited as the ability in which students experienced decreased growth, but was not identified by any student as an ability that decreased in importance. This is likely symptomatic of poor team experiences in teamwork or group decision-making. Excerpts from one student’s response to the assessment instrument’s essay components are shown below followed by the instructor comments. Area of most significant personal growth Leading others Describe how you have grown in this ability. “During the first semester, I ended up doing the majority of the work because I didn’t allocate enough work to other group members… by the end of first semester, I was burned-out because of all of the work I was doing and the other team members wanted to have more responsibility over certain areas of the project. It was at that time I realized that this IS a TEAM project. I was taking too much control and I needed to trust in my fellow team members.” Discuss how your growth in this ability has proven valuable (to you or your project). “I realized that I needed to work WITH the other team members’ strengths. For TEAMMATE1, that was giving him tasks that involved prototyping and hands-on- building… Once each team member had a specific area that they were they expert in, team productivity went through the roof!” Describe how this growth experience has prepared you for future professional development. “This experience will help me in the future because it has prepared me to work with others that don’t approach engineering in the same way I do. I will need to be open to trying different methods so that the number and quality of design solutions is maximized. It is important for a team leader to recognize the strengths of each team member and utilize them throughout the project...” Instructor feedback I agree that your growth as a leader was the primary area of accomplishment that I recognized in your work this semester. You have enabled your teammates to achieve at levels that they would not have reached on their own… The student’s work was scored as competent for growth description, proven value, and future development. See Figure 1 for the scoring rubric where the range of novice through expert corresponds to 1 through 5. This example characterizes an above average piece of work compared the performance of most students. A scoring summary is shown in Table 3. Table 3. Instructor means and standard deviations for scoring Professional Development Achieved Instructor ratings SD Growth description 3.4 .6 Proven value 3.2 .8 Future development 3.4 .8 Future Work The TIDEE team is continuing to gather data from professional skills assessments at several institutions. Data will be analyzed independently and across the progression from professional development planning, professional development progress, and professional development achieved. Results will inform the validity of the instruments for measuring student achievement in professional development and as ABET criteria measurement tools. Additionally, an inter-rater reliability study is ongoing to measure the consistency in assessment usage and a post-assignment survey is being conducted to measure student perception of the assessment. The TIDEE group is currently developing curriculum modules to support the use of TIDEE assessments. These will provide faculty with resources and lesson plans that prepare teams or individuals for successful use of the modules and will be organized to allow instructors to better integrate TIDEE assessments into project courses. References 1 Transferable Integrated Design Engineering Education Consortium. www.tidee.org 2 R. Gerlick, D. Davis, S. Beyerlein, J. McCormack, P. Thompson, O. Harrison, and M. Trevisan, “Assessment Structure and Methodology for Design Processes and Products in Engineering Capstone Courses” Proceedings of ASEE Annual Conference, 2008 3 Davis, D., J. McCormack, S. Beyerlein, M. Trevisan, H. Davis, R. Gerlick, P. Thompson, S. Howe, P. Leiffer, and P. Brackin. 2010. "Assessing Team Member Citizenship in Capstone Engineering Design Courses,” IJEE Special Issue on Applications of Engineering Education Research 4 D. Davis, S. Beyerlein, and I. Davis, Deriving Design Course Learning Outcomes from a Professional Profile. International Journal of Engineering Education, 22(3), 439-446 (2006). 5 http://www.boeing.com/educationrelations/ View publication stats View publication stats