Training Tips #2 Formal Assessment

This Article is part of a series of Training Tips, intended to help experts without an education background design and teach courses.

#1 Where to Start

#2 Formal Assessment

#3 Finding the Right Level

#4 Creating Exercises

#5 Content Delivery

This is the second in a series of tips for experts who also train as part of their work. I am aiming to avoid the jargon* used by the educational industry where it is not helpful to the technical expert as trainer.  

In the discussion around Tip #1, I said to “Start with Assessment”, by picturing the assessment of the set of skills your course provides. But, does this mean that I formally assess students at the end of the class? At the beginning of the class? Hang on, it’s a bit complicated. And, these decisions directly impact the course time, structure, exercises, and content.

The best way to assess people is to have them perform skills in a realistic setting that are as close as possible to the conditions that they will have on the job. I also believe that I would learn more about gaps in a learning program if I had a before and after assessment. [I will discuss levels of knowledge to be assessed in the next Tip.]

Some students want ‘certificates’ that they can have/display after the course and some don’t care about them. Some (most) students do not wish to be formally assessed.  

I, of course, wish to help as many students as possible, and get paid for it. My training competes with other training offerings. My training competes with both the supervisor’s attention & budget and the student’s attention. 

Right up front, let me say that my comments here are not about compliance training given for legal reasons. In those cases of course you must have formal assessment. If my training were mandatory, I could impose whatever assessment I wanted, but that is not the case.

Into this set of conflicting desires around assessment, there must be some compromises among the many possible assessment options. Here are my observations and preferences:  

Pre-Course Assessment

Formal pre-course assessment is not a help in securing funding or participation. Most supervisors are not going to change their opinions of my course based on a change in knowledge I measured during the course. They have already determined that their employees should take the course before they signed up. If they come back to work enthusiastic to try out their new skills, that will form the supervisor's opinion. 

Formal pre-course assessment sucks the energy out of the first day of the course. A quick, written quiz is not a very good assessment. Trying to complete a realistic workplace simulation assessment on day one when you don’t know how is embarrassing.  

Of course, as an instructor, you need to have some idea of the levels of students in the course to know at what level I need to teach, what the range of knowledge is, and who may be exceptional (high or low) in the course. I start each class with ‘introductions’ from each participant. I ask them for their job title, years with the company, and a bit about themselves. If it appears that they have some experience in the subject matter, I ask them aquestions to judge the depth. I ask them individually what they want to get out of the course during the introduction section of the course. This results in a fair understanding of the general starting point for most students.

An instructor wants to identify anyone who can be called upon to help in the course, with stories, as well as concepts. A smart, experienced student can be a resource if tapped, and can become a problem if not tapped. For example, they may feel the need to demonstrate their experience to the other students, which can get the course off-track. Or, they may drop out of the course mentally, or physically.

It is also helpful to discover students early in the course who are not ready to take the course and may struggle. An instructor setting the proper stage for the course, by for example being a good listener, will almost always discover such students during an interactive introduction. This has been rare in my experience.

Final Assessment

Here are four options for final assessment, my preference from worst to best:

Participation Certificate

This is the most common practice and if valued by certain people, especially in certain parts of the world. I refuse to give out participation certificates. They are literally not worth the paper upon which they are printed. A person should not be rewarded for sitting in a room.

Test at the End  

This would be the most common assessment technique using written or demonstration test. The more realistic the test, the more classroom time it will take. Passing the test gives the students a sense of accomplishment. However, in my experience, the risk of not passing taints the whole course. “Will this be on the test?” will be asked multiple times. Students will pay more attention, but will be distracted from thinking about how to use the knowledge at work and focus more on the test. 

Also, what to do with those that don’t pass? No certificate, when everyone else gets one? I did hand out participation-only certificates to those that didn’t pass the final assessment. Failure needs and deserves to be accompanied by coaching. This is difficult at the end of the course when everyone is otherwise waiting to leave. From a selfish point of view, if the instructor gets bad ratings because a student fails and is angry about it, telling other prospective students, there is going to be less work for the instructor. And failures might need to be reported back to the supervisor. This caused a bad reaction from some supervisors who do not want to hear about their employees failing tests, because they do not believe such failure is representative.

For all these reasons, I rarely gave tests at the end of the course. 

Observations During the Course

If the best assessment is supposed to be a performing realistic skills in a realistic environment, then perhaps the best assessment is identical to supervision in the workplace. It is not difficult for the observant instructor who spends a few days with students to be able to qualitatively judge their uptake of the material. I made the statement early in the introductory courses that I would not do a formal assessment, but I would report exceptionally bad performance and exceptionally good performance of students back to their supervisor. In more advanced courses with experienced students, I rarely had ‘bad performance’, and I was very pleased to report exceptionally good performance back to supervisors. I did give a course to mid-career engineers that was intended to be a refresher on various topics with updated information. The class was built around multiple exercises that made it fairly easy to judge individual performance. I added a whole day to the end of the course to provide one-to-one coaching sessions including my evaluation of their performance. I also provided feedback to some supervisors, informally. This type of assessment was very appreciated by almost all students.  

Example of a certificate for a course with a wide variety of topics and skills

In such cases, I proudly provided each student with a certificate of achievement, listing a number of skills they had demonstrated during the class. I signed each one in front of the student and told them what a great job they had done. Except, there was the occasional student in the class who performed poorly. In those cases, I had already taken them aside multiple times and coached them on the issues. When I gave them a nearly-identical participation-only certificate, they already knew they had struggled and were prepared for such a result.  

Most group exercises can be designed to provide individual observations. Ask questions of individuals to test knowledge. Ask for demonstration of skills from individuals during the exercises. Take notes. Provide timely feedback and coaching.

Especially with new courses, such observations during the course will be critical to tuning the course content and exercises. Naturally, make notes as you go. I often stayed after a day of teaching to make changes to content and exercises for the next course.

Workplace Assignments

In designing virtual courses there is a challenge that students will be too isolated. Consequently, in virtual courses you should come up with exercises for students to work with other students. Further, if there are ways to work with their own organization, that is great too. I extended this concept to assessment. Have the student return to work after a physical class (or at work during a virtual class) and create a work product just as they were taught in class. This could either be a work product assigned by their supervisor, or created just for the course, for example a complete design, analysis, interpretation, report, audit, etc. The instructor needs to skilled and comfortable reviewing such work products, and will need to also coach students until both are satisfied with the final outcome. I found this type of assessment to be very rewarding because in most cases the outcome generated both a valuable work product and a practitioner ready to create more. I made it a practice to stick with any student until they were successful. I provided very nice certificates of achievement listing high-level skills they had demonstrated. A few students needed substantial help to complete this assessment. It requires a lot of focus by the instructor to avoid doing the work for the students. Students who got this far in the course almost always completed the assessment and thus passed. Supervisors were very satisfied with this type of assessment. The main drawback is the time it takes the instructor, which could be substantial for a large class. This could be mitigated by getting assistance from other coaches in the organization.  

The instructor might have to be creative if the course material is not revolving around skills in the workplace. For example, a short course on a narrow topic may only be to provide knowledge about a topic, with no expectation for the student to apply a skill. In that case, I suggest creating a skill that demonstrates an appropriate use of that knowledge. For example, carry out a conversation with the instructor (or even better an expert from the student’s own organization). In one such assignment, we sent the student to the gas lift valve shop to interview the person in charge and to report back. The students were given several questions to ask during the interview. Not only did this help the students, but the person in charge of the shop created an excellent tour of their facility, permanently improved their communication with the engineers, and elevated his standing among them. Winners all around!

Ideally, a successful student would be able to create a portfolio of completed work products, demonstrating their competence. That is the best form of certification.

Assessments are perhaps the most challenging aspects in designing and providing a course to students who are also clients. I expect there will be a variety of opinions. If you have any suggestions, please provide them in the comments! I’m very happy to learn too.

If you have suggestions for future topics, I would love you get them. Thanks!

*If you need to discuss the assessment topic with educators, you may find this document helpful

I will try to use these hash tags for this series of articles: #trainingtips #training #learning #engineers

Burney is an almost-completely retired global consultant engineer, and Director of Retirement Testing at the Waring Retirement Laboratory