Low Impact Living: Green Prefab Coming to a Market Near You

“Effective GMP” discusses specific GMP topics useful to practitioners in compliance and validation. We intend this column to be a useful resource for daily work applications. The primary objective for this column: Useful information.

Reader comments, questions, and suggestions are needed to help us fulfill our objective for this column. Please send your comments to journal coordinating editor Susan Haigney at shaigney@advanstar.com.

EDITOR’S NOTE

A previous issue of “Effective GMP” discusses “Effective Training” (Journal of GXP CompUance, Summer 2009, Volume 13, Number 3) and the following key points that should be considered in management of a GXP training program:

* Training policy, standards, and procedures documented

* Training process strategy and approach defined

* Principles of adult learning theory considered

* Training needs analyzed and prioritized by risk analysis

* Collaboration of affected groups with defined responsibilities and requirements for each group

* Trainees and their organizations are “customers” of training

* Training appropriate for task

* Training materials and methods appropriate and effective

* Qualified training personnel

* Training performance

* Training effectiveness monitoring and maintenance

* Change training if needed

* Training documentation

* Efficient and cost-effective training

* Senior management support training.

Another previous issue of this column (Journal ojGXP Compliance, Spring 2010, Volume 14, Number 2) discusses several reader questions on adult learning principles including whether adults learn differently than children, the effect of age, and other issues related to adult learning.

Considerations in these articles are relevant to the training session in which the training content is presented. The following fictional letter to a training director discusses the numerical evaluation of training sessions and the interpretation of these evaluations – the accuracy, precision, type of data, skewness of the data, and the many factors influencing the data. A proposed evaluation form developed by the author is presented.

The training session is often the first visible and measurable step in the training process. It is undoubtedly a key component in the organization’s training program. Compliance and training professionals should be mindful of the points discussed in the following and apply them as needed in their respective training programs.

June 1, 2010

Dear Mr. Training Director:

I have tried to avoid responding to your request for feedback on the training evaluation statistics because I am not sure how you will respond to my comments. However, this is the second or third time you have asked – so here goes. This response is what I would tell a client if this were a statistical consulting job. Here are just the facts and my unvarnished opinion. in our site adult learning theory

I must tell you that after 20 plus years of training, I really don’t agree with numerical scores for evaluation. I know this must sound odd from a statistician, but let me expand on this. My thesis is that while numerical scoring for evaluation of training is very wide spread and even a pervasive required activity, it is not the best way to evaluate training. It is actually a barrier to improvements. Numerical scores don’t provide the information needed to improve the course.

First, let’s consider the statistics. We are working with small sample sizes of 5 to 35 attendees, each group is different, and scores are floating – there is no “truth” or absolute standard for comparison. As such, accuracy cannot be measured without a basis for comparison. This is the same problem in BioAssays. There is no standard and thus accuracy can’t be determined. The attendees could be comparing to their expectation for just this course, for other courses like this one, for all courses taken in the past, for the one outstanding course they took years ago, or for some idealized course given by an idealized trainer. There needs to be an evaluation of the course that is independent of the attendees past history, knowledge of the subject, and other courses taken. Of course this is not possible.

Next, scoring scales of 0-5 or 1-10 imply a false sense of accuracy and precision. Statisticians recognize these scales as “ordinal” measurement scales with the differences between scores as not being equal and not the same from attendee to attendee. The 0-5 scale is not different from A, B, C, D, F. The whole scale is “floating.” Each attendee has a totally different basis and background for scoring. A novice and an experienced attendee will have totally different scoring.

This type of data from a course is not normally distributed and is usually skewed to the left. That is, there are a few very low scores with most scores being higher. Further, averages are highly effected by only a very few low scores. An average that is found with a zero or 1 will always be low. Also, statisticians recognize that the average is incorrect for ordinal data. The median is the correct measure of centrality.

Comparing an individual course to a database of scores is a real apple and oranges comparison. A score of 6, 7, 8, or 9 may be a high score for some people depending on their background. Some people cannot give a score of 10 to any instructor no matter how good. The written evaluation many times will say “an excellent course” or “excellent presenter” and the numerical scores are 7s and 8s. I note that Olympic scoring “trims” the scores before taking the average.

Next, let’s look at some human factors. And forgive me for being blunt, but I have thought about this a lot over the years. Do we expect all of the trainers and speakers to be above average? Are we willing to tell almost half of all trainers and speakers they are below average? Nobody wants to be below average. What about volunteers? Will they continue to volunteer if they are rated below average very often? Numerical scoring will cause hard feeling among the speakers. Trainers are being pitted against other trainers for no good reason. It doesn’t matter what the numerical scores are. Did the course meet or exceed expectations? Did the attendees learn something? The focus needs to be on improvements, not scores. A numerical score will never tell me what needs to be improved or how to improve it.

I have found that there are many factors that will influence the numerical scores. Here are some I have encountered. Was attendance required? Are the attendees mad because they had to attend? I have seen a supervisor or manager require their staff to attend a course on the theory that they “will pick up something useful,” even if they don’t understand the subject being presented. Needless to say, these attendees will universally give low scores even if the trainer is outstanding. Attendees will often score a course low to “get back” at the person that made them attend. Attendees who don’t understand the material at all and are totally lost will give a very low score. see here adult learning theory

Was the room comfortable, hot, or cold and drafty? Was the room crowded? Was it hard to see the screen? Does the trainer have any control over the number of attendees, the room, the crowding, the temperature, the lighting, who attends, the food, etc.? Why should that be part of the trainer’s evaluation?

Did the attendees see an announcement before attending? Did they know what to expect? Did the correct group attend? Courses are designed for specific audiences. People with a strong background in the subject want an advanced course. Those new to the subject want an introductory course.

The personality of the presenter is a huge factor in scoring. A lively and humorous speaker will always score higher even if the correct material was not presented. In our society, the “sizzle” sells better than the steak.

In filling out the forms, if there are too many questions, the attendee will just “run” down the page checking off the same number for all questions. Instead of getting 15 or 20 independent considered estimates, the attendee has given one estimate 15 or 20 times.

Attendee participation is highly dependent on the culture of the company or the industry. People are afraid to speak up in a group they don’t know from the same company. They are more willing to speak up in a group they don’t know from different companies on the theory they won’t see them again soon.

So, there it is for what it is worth. I am attaching the evaluation form that I use in my classes. If more than 10% of the attendees say the same thing, then I would consider making a change in the course.

Sincerely,

Lynn Torbeck

Statistician

ABOUT THE AUTHOR