Introduction

Continuous Improvement (CI for short) is the process of continually analyzing the performance of some aspect of operations, and then applying changes to improve that performance.  It is a critical component in maintaining the health of any operation. In the maritime training environment, the focus of CI should be on improving the effectiveness of training, with a secondary focus on improving efficiency.

This topic is especially timely right now. Vessels, equipment and job routines in the maritime industry continue to become more complicated and sophisticated. Deeper knowledge and more specialized skills are required to operate safely. However, familiarization training in most organizations has not changed much. There is an increasingly large gap between operational training and the knowledge needs of modern seafarers. A program of continuous improvement for operational training is a necessary tool to close this gap.

This is the second in a series of articles intended to give some practical tips for using CI to improve training effectiveness and efficiency. Implementing a modest CI process for your in-house training is neither expensive nor difficult. Even a small program can produce a tremendous ROI in safety, efficiency, trainer engagement and trainee satisfaction.

Our first article provided an overview of CI, talked about why it is important, and introduced Key Performance Indicators. This post continues by talking about specific key performance indicators which can be used to measure maritime training performance.

Key Performance Indicators

Key Performance Indicators (KPIs) are the foundation of any CI process. They are measurements used to evaluate effectiveness and efficiency. This is critical –  unless you can measure whether changes have created an improvement, you’ll have no way of telling whether you are moving closer to your goal. KPIs provide that information by measuring the success of every change you make.

KPIs should satisfy the following requirements:

1. They must be aligned with corporate goals.
2. They should react reasonably quickly to changes in training. When we make changes, our KPIs should be able to tell us whether they were effective or not.
3. They must track something that is “measurable”. Some important things can be difficult or impossible to measure.
4. They must track something you have some control over. Otherwise, although they may be interesting to track, they are not helpful in the process of tuning training outcomes.

As with many data-driven systems, technology is your friend here. If aspects of your training activities are delivered and managed with a learning management system (LMS), you can use that to report on metrics about the learning process. In fact, metrics are one of the greatest benefits of an LMS, however, they are only now starting to emerge in maritime familiarization training. Therefore, this post will first focus on KPIs which can be obtained without the aid of LMS (“Manual” KPIs). The next article in this series will discuss KPIs possible with an LMS (“LMS-enabled KPIs”).

There are many good resources which provide information on KPIs and frameworks for evaluating training. One that I have found to be useful is a paper written by Deniz Eseryel called Approaches to Evaluation of Training. It’s short and a bit academic, but sure to give you some good ideas and additional background.

Another excellent article to read, Training Evaluation Framework Report, presents a framework for evaluating training. Although it addresses the training of social workers, it is an excellent article which applies to any kind of training.

Let’s now turn our attention to maritime training KPIs.

Manual KPIs for Maritime Familiarization Training

There are many KPIs which we can track to evaluate changes we make to familiarization training as part of our CI process. These fall generally into four categories:

1. Ask the trainee
2. Ask the trainer
3. Evaluate the trainee
4. Record performance

Let’s look at some KPIs in each of these categories:

Ask the Trainee

Our students can be an excellent source of information. The signal to noise ratio can sometimes be quite low, but there will be good data there.

The most common and obvious technique is to have the trainees fill out an evaluation of the trainer and the familiarization process. This evaluation can be:

1. Formative: used to inform the trainer and help them improve their technique.
2. Summative: used by training administrators as a means of trainer and program evaluation. This data will be useful in terms of hiring and advancement, as well as in providing suggestions to improve a trainer’s teaching. They also help on a company-wide basis to optimize your train-the-trainer program and the familiarization process.

Student evaluations are easy to implement and give you excellent insight. A short resource with tips and examples for creating trainee evaluations can be found here.

Another very useful, though rare, technique is to revisit the training evaluation 6 or 12 months after the completion of the training event. A seafarer will have gained some on-the-job experience and have had time put the knowledge and skills into practice. They may be able to provide a more informed and useful evaluation of the training. For example, they will be able to answer questions such as “What was missing from your familiarization training that you now believe should have been covered”. Or “What was covered during familiarization which you feel was not relevant to your position as a deckhand”.

Ask the Trainer

It is much less common to gather trainer feedback. This is a shame as it can be very insightful to ask the trainer to reflect on the process during each training event, or immediately after. There may be useful ideas which are fresh in their head, but diminish in immediacy over time. Like trainee evaluations, feedback can be collected through forms.

Trainers can be asked to comment on a variety of metrics such as: the preparedness of the trainees, the appropriateness of the duration of training, the quality of the materials used to support the trainees, the quality of the assessment techniques, and so on. This last question regarding assessments is very important. At BC Ferries, where I am involved in, we get a great deal of feedback on the individual questions used to evaluate trainees. This feedback is critical in the process of continually refining the assessments themselves.

Another useful technique here is to ask the trainers to perform self-evaluations. Ask them if they feel they performed well, what they could have done better, what new techniques they tried and found to be successful, and what support they require to deliver a better experience next time. In order to get honest and useful answers, it should be clear that this information will never be used to evaluate their performance and will only be used to support them and improve training throughout the organization. An additional benefit of a self-evaluation is that it causes the trainer to reflect on the process of training and to make them a partner in its improvement.

Evaluate the Trainee

Candidate evaluations can also be used as a KPI.

A couple points on this. It is a good idea (and a common) to perform assessments not only at the end of training, but also at the beginning of training. There are many reasons why: first, if you provide an evaluation at the start of training, you are aware of what the candidate does and does not know. By comparing that to the results of evaluation conducted at the end of training, you can determine how successful training was. Secondly, a pre-assessment also helps you improve hiring practices and pre-familiarization self-study materials. Finally, pre-assessment results can be used by the trainer during familiarization in order to tune his or her program to the candidates in hand, and to ensure that critical gaps in knowledge are covered.

Another good technique is to re-evaluate the candidate after some passage of time. This provides information on how well knowledge is retained over time. If critical knowledge is lost, then the training program can be altered to reinforce this knowledge and to recommend further training to individual candidates.

Finally, use caution in applying these metrics without analysis and thought. Assessment outcomes can vary significantly depending on the method of assessment, the person administering the assessment, the differences in the candidates, the age of the assessment itself, and more. Always think carefully about whether (or how much) a KPI is being influenced by actual training effectiveness.

Record Performance

The final category speaks to the need to record performance in terms of safety and efficiency. No doubt you already have a system in place for measuring key aspects of operation performance. You can mine these other non-training KPIs for information related to training performance.

One simple example is a KPI which measures the duration of familiarization training. Some organizations do not have fixed training agendas and vary the number of days of familiarization. Keeping good records on this metric can provide very useful data. For example – has the number of days crept up over the years? Why? Does it vary according to the trainer? According to the number of trainees? According to the vessel? The answers can provide insightful clues on how to improve outcomes.

Another example is the mining of safety-related incidents. Statistics on these should be visited regularly to uncover recurring issues which can be addressed through a modification in training. Likewise, for other operational metrics such as turnaround times, customer service complaints, and so on. There is a wealth of information you either are already collecting, or begin collecting that will help you improve training, and subsequently safety and operational efficiency.

LMS-enabled KPIs for Maritime Familiarization Training

All of the KPIs spoken of above can be collected and analyzed manually. However, if you use a learning management system to deliver familiarization and job training, you’ll likely also have access to a variety of other deep KPIs which can provide insight that is simply not otherwise possible.

In the next installment of my blog, I will discuss examples of LMS-enables KPIs. I think you may find it very interesting (I know I do), especially if you are not already aware of the kinds of learning metrics that LMSs can provide. Be sure to follow this blog if you would like to be informed when that next installment is available.

In the meantime, please feel free to comment:

1. Do you have a CI process in place for training?
2. Do you use or can you think of other useful training KPIs?

Please let us know so we can learn from your thoughts.