February 21, 2008
Presentation Measuring and Improving Training Effectiveness

The presentation is available as:
a Powerpoint file (312 kb)
or a pdf (255 kb)

The resources discussed during the presentation are listed below. So that you can read reviews and descriptions of each book , clicking a title links to the Amazon.com book description.

If you are looking for used copies http://www.fetchbook.info/ is a good site to find them. Use care when ordering used books, especially if you aren't familiar with seller rating systems and other ways to help safeguard your purchases.

The meeting notice:

An article in Quality Progress magazine a couple of years told us that the most common non-conformance in ISO 9001:2000 audits was, "failure to establish an adequate method for measuring the effectiveness of training".

Our session for this month covers a topic that at once seems familiar and feels like it ought to be simple, but also causes some of the biggest headaches for many of us "quality types". Most quality standards and customer requirements demand that we have good measures for the effectiveness of our training and that we tie our training to our continuous improvement efforts. Somehow, even though measuring and improving may be our strong points in other areas, this particular area presents a difficult hurdle for many.

The presenter:

Susan L. Reynolds, CQE, CSSBB, CQA, CMQ/OE, CIA  
(Note CIA = Certified Internal Auditor)

Susan is our web master and news “distributor” for ASQ Section 105.

She has 30 years of experience in Quality Assurance, Process Engineering, Systems Management, and most recently Program Evaluation.

She has developed complete quality systems from the ground up, led internal audit teams, audited many, many suppliers, and was a traveling ISO auditor doing interdivisional audits in one large corporation. Susan also worked as a consultant for a little over one year teaching DOE courses around the country.

Susan’s current role is as an analyst for the Maine State Legislature in the Office of Program Evaluation and Government Accountability.  She was lead auditor for two of that office’s reviews, Information Technology and Economic Development (reports can be found here.)

Through most of her career she has had some degree of involvement in training programs; as a trainer, as a quality systems auditor, or when assessing the effectiveness and efficiency of programs. Each of these roles has involved designing, assessing or recommending corrective actions for the measurement of training effectiveness.

---

 Have you ever heard this response (or an equivalent) to corrective action requests?  "It was operator error, but we'll tell them to do better next time"
... and you may have begun to wonder, how many times in a row should it really take for someone to correct a problem obviously deeper than an error by any one operator? Perhaps you too have been applying the "whack a mole" approach to training by attacking each problem as it rears it's ugly little head.

Hey, maybe a more systematic and effective way of measuring and improving the effectiveness of training could help!

After  almost 3 decades of QA work; internal and external auditing, developing quality systems and reviewing corrective actions written by others as well as  developing my own,  I have found that one of the most common true root causes for problems boils down to an ineffective transfer of knowledge and/or skills to people at many different levels of the organization. Yet the most frequent claimed "root causes" and  "corrective actions" I ran into were just creative ways of saying the same thing over and over: "It was operator error, but we'll tell them to do better next time".

The operator would get a bit of training if he was lucky or a stern lecture if he was not. A few others in the work area might get a heads up as well, and things would look up for awhile. Eventually a very similar (or even identical) problem would occur and lo and behold, the root cause and corrective action were deja vu all over again.

So we applied the usual band aids. We trained and trained and trained. We developed measures for our training that looked like the others we had seen. Every training session was followed by an evaluation form and the results were duly collected and reviewed. We seriously contemplated their meaning and made adjustments. There were regular short term improvements that encouraged us to plod on.

... but then over time as we stood back and looked at the big picture results of another year from our corrective action system we found that "operator error" was, in one form or another, STILL the most commonly listed root cause reported. When supervisors were pushed to find and report and more true, fundamental root causes there were variations to the theme but in the aggregate they generally came down to training; the operator did not have all of the knowledge or skills that were needed to make the decision or perform all of the tasks required in their jobs.

It was not a problem we failed to notice. We saw it, commiserated about how prevalent it was and tried to fix it.

Awhile back I decided it would be a really good thing if I could find a way to stop using the "whack a mole" approach to Measuring and Improving training  effectiveness and I made a concerted effort to find real best practices in order to pull together a training quality system that would make use of my decades of knowledge, my wicked impressive certifications and all of the best practices I could assemble.

You will be handed the fruits of my investigation and study at our February session on the 21st, including step by step methods, some spreadsheet files with examples of tracking systems, and a list of oodles of resources of what other "wicked smaht" people have to say on the subject.