Web Based Training and Levels of Evaluation
But what are we talking about when we discuss evaluation? I, like most of us was taught about Kirkpatrick's levels of evaluation. Don't know what that is? Click here to learn more.
Level 1 is the easiest evaluation type to capture in web based training as almost all authoring tools include some evaluation tool that is capable of such. Keep them short to maintain as much accuracy as possible. If this type of evaluation is too time consuming, learners may arbitrarily check off all the same grading for each question thereby nullifying the results.
Level 2 evaluations are also not overly difficult to create in most authoring tools. Usually they are achieved much the same as the level 1 evaluation except that a grade can be assigned to the results. Several things to keep in mind are:
- Be sure the evaluation addresses the learning objectives of the course. Don't just ask questions for the sake of asking questions
- If possible, use a randomize function to rearrange the order of the questions and the order in which the possible answers are displayed. this will make each version of your quiz different from the last
- Choose a small number of questions from a larger pool of questions. Again, creating a unique quiz each time the course is accessed
- Make all possible answers appear plausible so the correct answer isn't easily guessed
- Avoid True/False questions for the same reason as above
Now here is where things get tricky for evaluation of web based training. Level 3 can be one of the most costly types of evaluations for most organizations, especially those that have operations in multiple locations across the country.
Here is one suggestion. After a week or more has past, send an email to each student's supervisor. If your LMS can generate this automatically, all the better. Ask the supervisor to observe their employees for demonstration of the new skills in the work place. This assumes the supervisor has taken this training or is equally knowledgable about these new skills. Have the supervisor report back on the results. Once the data is collected you can compair results and determine if the training has had an effect on the behavior of those that have completed it. Hopefully the supervisors and the business as a whole are willing to buy in on this type of participation, otherwise you may have to perform this fieldwork yourself.
Level 4 does require a commitment to follow up on your part, however the organization is likely already generating reports on results that effect the business. Some examples of these reports are revenue, costs, number of workplace accidents, etc. Make sure that you drill down to a level that matches the objective of the training. This will ensure that some other area of the business isn't responsible for any changes and you can therefore show return on investment.
In both level 3 and 4 you may want to also capture this data when you are designing your training. This way you will have a base line to compare your later results with, but you will also learn where the performance gaps truly are.
Multiple Intelligences
- Linguistic
- Logical-Mathematical
- Visual-Spatial
- Musical
- Interpersonal
- Intrapersonal
- Naturalist
- Bodily-Kinesthetic
Here is an interview with Dr. Howard Gardner explaining Multiple Intelligences far better than I could:
One could incorporate this into their lesson design by having the students select from a list of activities that they can perform to complete that particular lesson.
Example: If the topic was the use of propaganda during World War II, students could select one of the following tasks:
- Read an article on propaganda and explain it to the class
- Create their own propaganda poster
- Write a song that attempts to convince others of an idea
- As a group shoot a video skit on propaganda
Pay Attention: Digital Learners
Every instructional designer and training consultant should watch this video.
Every Instructional Designer and Training Consultant should watch this short video. Generation Y or the Millennial Generation is entering the workforce right now. Much of the information is already out of date but the importance of this video remains just the same.
Did you know?
Organizations provide an average of
- 30 hours of training per employee
- 37 hours for technical employees
On average, 564 employees are served by one trainer.
Older and/or senior employees receive the most training.
The most used training practice in the workplace is tuition reimbursement plus paid conference attendance.
The 100 best companies spend the most on training and employees.
Learning 2.0
We take it for granted but we are only in the third decade of the personal computer revolution. Perhaps less if you consider that the personal computer really didn’t become the mainstream medium that it has become until the mid 1990s. Certainly computer based training and later web based training have only been around for less than two of those decades.
Of course the twenty first century really changed things. Better operating systems showed up, while faster and more efficient microprocessors drove the industry forward. Multimedia really began to affect online training with the introduction of high speed internet connections. Now you could bypass the previous limitation of dial-up and stream full quality multimedia over the net. Also the current generation that has began to enter the workforce, have expected a level of technology that they had for their entire lives. Unlike my generation, they could not remember a time when there were no personal computers.
So how do we satisfy these current crops of learners? I think we all know that successful training needs to be collaborative to truly be engaging. So how do we make web based or online training collaborative? For that matter how do you incorporate new technology into classroom training for more than just the sake of the technology? Whether you call it Web 2.0 or Cloud Computing or Learning 2.0, these new technologies may be the answer. When the World Wide Web first became available to the masses, what did it consist of? Well mostly it was just page after page of words and photos. There was very little interaction while you were online. At best you were an observer. Sure you could download software, but it only could be executed once it was on your PC.
Think about the following technology based services:
Each of these services remains on the internet (you don’t need to download them in order to use them) and they all are very collaborative in nature. With the social networking sites like Facebook, Myspace, Linkedin, you connect with other users and share information with one another. Sounds like much of the group work we as trainers facilitate, doesn’t it?
Facebook doesn’t create the interaction but rather facilitates it, much like the instructor of a classroom session facilitates learning.
YouTube continues with the concept of social sharing and stimulates us to be more creative than if we didn’t have access to such a service. Although my still camera and my cell phones have had the ability to shoot short videos, I didn’t shoot video until it occurred to me that I could share it on
YouTube. I think at present teachers are using YouTube in the same way your Social Studies teacher used the Audio Visual department in school. Simply show you a film on whatever topic was relevant. I suggest that YouTube could be a delivery mechanism for the students to visually collaborate with each other. I’m certain each class probably has one or two Steven Spielberg’s with a camera phone ready and willing to take it to the next level if they were given the chance.
Instructors are often subject matter experts in the fields in which they teach. I'm sure many of us remember teachers from school who had personal collections that they would share with the class. Unfortunately when the class ended the teacher brought these collections home again. If you are a teacher of art or photography you certainly could make your collections available to students 24/7 by using sites such as Flickr or Webshots. This could be expanded as well to a method for students to share and collaborate their own work. It even could be a method for submitting work to the instructor.
Blogging is certainly a great supplement to the traditional lecture. Like most corporate trainers, I usually only had access to my students for about a week and then it was on to the next round of students. Wouldn’t it be great if they could read my latest lecture online? Remember that a blog is also collaborative. Anyone is welcome to comment on each entry that I place here and certainly those comments would affect what I write now and in the future. The possibility for discussion truly makes it collaborative.
A good use of sites like Wikipedia is certainly for research, but don't forget that this is a collaborative tool as well. Your students can write and edit articles on Wikipedia. The concept isn't limited to Wikipedia as well. Students can setup and maintain their own online wiki, using it for class projects or assignments, each one contributing to the whole and sharing ideas in the process. Another piece of the puzzle are the portals like iGoogle and netvibes. Nothing personalizes a web experience than having all of your favourite stuff contained on a single start page. Where once, sites such as Yahoo and MSN attempted to predict what you wanted to learn about or see online, now you get to decide what your web experience looks like.
Of course the trick is getting all this into our training. I know that my focus moving forward is to use as much of the technology as possible. We are really just at the gates of the potential of learning with technology, and I am only scratching the surface here. Here is one piece of advice though; don’t decide on an online service and figure out how you are going to use it. Instead figure out what you want to accomplish, and then look for a collaborative online solution. It will certainly be interesting to see what we have done with it in another three decades from now.
Principles of Adult Learning
Those of us in the training profession often refer to the principles of adult learning. We talk about it to our stakeholders and subject matter experts, put it on our resumé, and mention it to each other in training discussions.
Do we know what those principles are? Over the years, I have attended many sessions that included the so-called principles of adult learning as part of the course, and I have seen similar yet unique lists of these principles for almost every course that I have been on. In most cases, we turn the page in our workbooks, and that's the last place it's mentioned. Here are some examples from my past that I have gathered up in one spot for comparison.
The local college where I took an Adult Teaching course used this list:
Adults must want to learn
Adults will learn only what they feel is a need to learn
Adults learn by doing
Adult learning centers on problems, and the problems must be realistic
Experience affects adult learning
Adults learn best in an informal environment
Adults want guidance, not grades
A company my former employer hired had this list:
Adults like active participation in their learning experiences.
Achieving goals is the adult learner’s responsibility.
Adults have a wealth of experience and like to share what they know.
New knowledge and skills must be integrated into previous learning. Information that conflicts with previous knowledge or values will be integrated more slowly.
Understanding how learning will be useful (WIIFM) motivates adults.
Learners need to practice performing a skill or solving a problem.
Self-esteem is a critical component of learning.
Adults learn better when they are having fun.
The energy level and interest influence attention span.
To maximize retention, adults need small chunks and frequent summaries of information.
Note: WIIFM = What's In It For Me (in case you didn't know).
Another such company had this list:
Adults need:
A clear introduction, including objectives, benefits, and an agenda
An opportunity to express themselves and share experiences
A variety of methods
The appropriate level of challenge
A comfortable learning environment
Immediate feedback concerning their progress
Here is a list provided by an individual hired to train us at the same company:
Adults want...
A sense of accomplishment
To save time
Adults bring...
Experience
Established behaviors
Adults expect...
Efficient use of their time
To know the end goal and the process
An American university used the following (I removed the subtext as it was rather long-winded):
Adults are autonomous and self-directed.
Adults have accumulated a foundation of life experiences and knowledge, including work-related activities, family responsibilities, and previous education. They need to connect learning to this knowledge/experience base.
Adults are goal-oriented.
Adults are relevancy-oriented.
Adults are practical, focusing on a lesson's most useful aspects in their work.
As do all learners, adults need to be shown respect.
Of course, I can’t talk about adult learning without mentioning Malcolm Knowles. He has written entire books on this subject. However, the following main points summarize what he has said:
Adult learners need to know why they are learning, what they are learning, and how they will learn.
Adult learners move from being dependent personalities toward being self-directed human beings.
Adult learners accumulate a growing reservoir of experience that becomes an increasing resource for learning.
Adult learners' readiness to learn becomes increasingly oriented to the developmental tasks of their social roles.
Adult learners' perspective of time changes from one of postponed knowledge application to immediacy of application, and accordingly, their orientation toward learning shifts from one of subject-centeredness to one of problem-centeredness.
Adult learners are internally motivated to learn by their intrinsic value and personal payoff.
After reading all these versions, I felt inclined to re-write it into my language, and in such a way, I might find it useful when conducting or designing training. Here is my attempt:
Paul Wilson's Principles of Adult Learning
Show your adult learners respect by providing a comfortable learning environment where training has a clear benefit, objective, and agenda.
Organize your training into easy-to-manage sections with clear summaries at the end of each section.
Adults bring experience to the classroom. Use this experience in your lessons. Adults can teach one another through group activity or discussion. Adults must see how new learning fits into their prior experience.
Adults will become motivated to learn when training can show value by providing solutions to problems they have a vested interest in. Adults will focus on those aspects and set their own goals for learning.
Remember that adults learn using various methods, and learning is retained when enjoyable. Adults are self-directed and cannot be told when it's time to learn.
Of course, like Knowles, I expect that I may go back and revise mine a little over time. He had over twenty years to consider his. I've just really thought about mine for a couple of days. Please let me know if you feel I've missed anything or got something completely wrong.
Dealing with Bad Behavior in a Classroom Setting
Have you ever had a learner in one of your training sessions who is displaying bad behaviour? It is rare in adult learning situations, however it can happen. Sometimes you are dealing with “Type-A” personalities such as sales executives who feel they have nothing to learn, or perhaps a long tenured employee who questions your ability to teach a subject they know well.
Have you ever had a learner in one of your training sessions who is displaying bad behaviour? It is rare in adult learning situations, however, it can happen. Sometimes you are dealing with “Type-A” personalities such as sales executives who feel they have nothing to learn, or perhaps a long-tenured employee who questions your ability to teach a subject they know well.
There are several small things that you can do while still maintaining respect for all your learners. Making eye contact with the learner who is exhibiting poor behaviour can sometimes be a gentle reminder that it’s time to get focused on the training. You can also walk toward that learner, shifting the attention of the class toward the general area where the learner is seated. This can have the effect of making them much more aware of the rest of the class. It will appear the class is looking at them when in fact they are simply listening to you. They also become aware of your presence when they realize that they are now within earshot of the instructor.
Years ago I took a course an Advanced Instructional Techniques from Langevin Learning Services, and the Course Leader Steve Flanagan taught us this technique for resolving situations in which learner behaviour is creating a problem for you or the class. When you identify that someone is disruptive to your training, and you have exhausted other more subtle methods to handle the behaviour, take them aside during a break and privately go through the following...
I (Your feelings)...
When you (their behaviour)...
Because (affect on you or the class)...
For example, if you had a student who was constantly late from breaks and interrupted the class in order to catch up on what they missed, you might say to them at the beginning of the next break:
“When you return late after breaks, I feel frustrated because the rest of the class is falling behind schedule.”
Steve suggested that constructing these “I-Statements” was beneficial because if you try to confront a learner displaying bad behaviour without planning what you might say, you can easily get trapped by emotion and blurt out something you may later regret saying. Having the statement prepared in your head beforehand will save you getting into an emotional debate with them. If the learner attempts to defend their behaviour, simply repeat the statement to add emphasis to what you already said. In all likelihood the learner will realize what they have done wrong and apologize, allowing you to continue to train the rest of the class.
Evaluating Training
Evaluating training is crucial for any organization. Without it, we cannot show if training is a worthwhile activity. The most widely used and accepted standard to evaluate training is to use Professor Donald Kirkpatrick’s levels of learning evaluation. The theory is that the more of these you use, the more complete your evaluation of training will be. Also achieving evaluation on the higher levels, while difficult, is a more effective measure of training success than just the lower levels which are easier to achieve. These levels are as follows:
Level 1 – Reaction
How did the learner feel about the training?
Level 2 – Learning
Did the learner retain the knowledge or skills taught?
Level 3 - Behaviour
Did the learner apply the new skills or knowledge in their daily life?
Level 4 – Results
Did the skills or knowledge have an impact on the organization’s results?
Level one is probably the easiest learning evaluation to achieve. In the training industry, we sometimes refer to these as smile sheets. These are typically handed out at the end of a session or coded at the end of an e-learning course to simply capture the feelings that the learner had with the training itself. It’s a well-known fact that many learners arbitrarily fill these out to either exit the class a few minutes earlier, or simply rush through them because they don’t really care about the data being collected. I always learned to take these with a grain of salt since I conducted training within an organization where my learners were mandated to participate.
One suggestion to make these more effective is to introduce an earlier level one evaluation to gauge what the training is like for the students so far. Perhaps if you are teaching a one-day session, hand out a modified feedback form after the first morning break. If it’s a multiple-day session, hand out the form at the beginning of day two and so on. You could ask your learners what they would like more of, what they would like less of, and what they would like to see that currently has not been addressed so far. You must be careful to preface this by suggesting to your learners that you may not be able to accommodate all suggestions as you must stick to the course objective. The reason you do this is to maintain control and not disappoint those who suggest something way out in the left field. Being able to change your teaching style or accommodate a particular learning style is hugely beneficial. It’s also beneficial to give your students time to complete feedback when there are no distractions such as getting through rush hour traffic a few minutes earlier.
A level two evaluation is a little more challenging than level one, however, most trainers include some type of test or quiz upon the completion of training. This should determine if the learning objective has been achieved. Most organizations use a combination of fill in the blanks, multiple-choice, and true-false type questions. Whenever possible a practical evaluation is preferred. For example; if you were teaching learners how to fill out a tax form, the level two evaluation should have them fill out the very same tax form. Answering some additional questions to address variations on how or why a form may be filled out a different way could also be beneficial, but training should concern itself more with doing as opposed to knowing or understanding.
Here is where evaluation gets more difficult. Level three often requires fieldwork and can be very labour intensive. If you trained employees all across the country to perform a certain task on the job, you need someone in all locations to observe and report back to the organization whether or not the task is being performed to a satisfactory level. Typically each location has a Manager who can perform such evaluations; however, there might be a margin of error created by Managers who do not align themselves with the organization’s objectives. When completed properly, a level 3 evaluation can teach you that there may be a question of adoption. The results can either be positive or negative, however negative doesn’t always mean that training was to blame for poor results. There may be a bigger issue related to the commitment that the organization needs to address.
Level four is also a challenge for many organizations. In all likelihood, the information mechanism for gathering data for a level four evaluation probably already exists. Proving that any lift in results is at least partially from training can be difficult to show. Of course, other areas of the business tend to want to snap up whatever credit for success there is for them. If a business launched a new product, often the marketing department will take credit for the improved results rather than allowing training to enjoy or share the success. Probably the only way to ensure that you attribute training success to the training is to ensure there is a test group or a control group. For example, if one group of employees took the training while another group’s training is delayed you could compare the results of the two groups to show how effective the training was for those who benefited by it.
Hopefully, this entry gave you some insight into how you can improve your organization’s methods to the evaluation of training. It really is beneficial to show the return on investment in training back to the organization and not just allow the smile sheets to be the only argument for why training is necessary.
Teaching Tools
I am taking a course at the local college on teaching adult learners. Today's session was on teaching tools. The study covered the traditional teaching tools, current and future educational technologies and selecting and utilizing the appropriate teaching tools. The main point of today's session was that training is really about the message. All the technology in the world can help you deliver the content, but obviously, it cannot replace the content itself. Our instructor mentioned a quote that I didn't get the author of; "A good aid is like a window; it should not call attention to itself; it should just let in the light." This made great sense to me and put the topic into context.
So what is a teaching tool? Simply put, it's anything that is used to assist in the delivery of training. Sometimes we use teaching tools to add variety to an existing lesson plan, while other times, it clarifies or enhances the lesson. Traditional instruments include chalkboards, whiteboards, overhead projectors, textbooks, flip charts, audio, video, PowerPoint, etc.
Selecting a teaching tool can be a difficult task. It's easy to say that I will choose a teaching tool such as a SMART Board to deliver my lesson. However, since a SMART Board can cost thousands of dollars, it may not be within your ability to provide such a tool. In addition to cost, some of the other factors that enter into selecting a teaching tool are considerations such as the domains of learning (cognitive/affective/psychomotor), learning styles (visual/auditory/kinesthetic), availability, time constraints, classroom size, facilities, your ability to use the teaching tool.
Another consideration is Edgar Dale's Cone of Experience, which suggests that we learn...
10% of what we READ
20% of what we HEAR
30% of what we SEE
50% of what we SEE and HEAR
70% of what we SAY
90% of what we SAY and DO
I’ve since learned that Edgar Dale’s model did not include percentage numbers as is often represented, and even Dale considered this somewhat anecdotal.
You will notice that Dale's cone of experience starts at the top with relatively passive learning and ends at the bottom with much more active learning. Ideally, you want to choose teaching tools that will bring you closer to the bottom of this cone to keep your students engaged and more participatory.
We spent most of the day discussing the various tools and how they could be utilized. I won't repeat all of these details, but to summarize, you need to consider each tool's strengths and any limitations and plan for contingencies.
For example, my group studied in detail the use of flip charts. We discovered that flip charts offer the following advantages as a teaching tool:
Easy to use
They are a great backup to PowerPoint
Excellent for group work
Relatively inexpensive
Easy to see
We considered the following disadvantages as well:
Difficult to use if you have poor handwriting
No spell check
Perceived as not as professional as other methods (low tech)
It can be awkward to work with
We came up with some tips for use as well:
Prepare your pages in advance
Group participation
Use a variety of colour
Make sure you have a method to display (easel or masking tape)
Make sure you have additional markers
These types of considerations should be made when you consider any teaching tool. Of course, something simple like flip chart paper probably does not require many reviews. However, using a laptop and LCD projector for a PowerPoint presentation may be challenging when you need to consider where this equipment will be, What room you will be training in, and whether or not the room even has a surface you can project onto. Make sure you spend the time asking yourself the advantages, disadvantages, and other considerations you should be making when developing your lesson plans.
First Entry
I previously had another blog up and running for many years. I switched from topic to topic as my feelings toward what was important in my life changed. I've come to realize that my blog should be about what I have been passionate about for the most significant amount of time. For me, that is training.
Training has been a cornerstone of all that I have done in my life. I may have been a manager or salesperson, but my common element is training. As a salesperson, the part of the job I loved the most was educating my customers about the potential products and services they could enjoy from my company. As a manager, I taught my staff everything I could for them to be better at what they do every day.
In recent years I have realized that training is what I should be doing. I have been an Instructional Designer and Trainer for over five years and have enjoyed almost every minute of it. To me, this has been the most exciting and rewarding career I could have ever asked for. When people ask me what I see myself doing in five years, my answer is always the same. Training!
To me, training is my passion and my life. What better topic could I follow as I continue my career as a training professional.