The following post is part of a series of articles on maritime assessments originally written by Murray Goldberg for Maritime Professional (found here). We have decided to re-post the entire series based on reader feedback.

Introduction

If you have been following along, this is the conclusion of the series on officer and crew member maritime assessments. This last article of the series is intended to “put it all together” and present a list of good techniques and practices (along with supporting arguments) that contribute to valid and reliable assessment. Last week, I presented tips one through three. I highly recommend you read that article before continuing onto this one. In this article I present tips four through six.

If you’d like to download and print a convenient one page sheet containing all 6 tips, please click here to download it.

Let’s get straight to it.

Tip 4: Train and Support Your Assessors

One of my readers recently sent me the following comment:

“The problem is getting skilled assessors with practical experience who can introduce a level of subjectivity in a non-threatening manner.”

Skilled assessors are partly made by experience, and partly made by training as assessors. Most maritime organizations use qualified and experienced mariners as assessors. Obviously this is a very good idea since they will have a solid basis for judgement. But just because they are experienced as mariners, does not mean they have all the necessary skills to become good assessors.

Some argue that this is one of the greatest weaknesses of current assessment practices in the maritime industry: trainers and assessors may be great mariners, but less adept at training and assessing. Fortunately, this weakness is also one that is relatively easily corrected. We can do so by:

1. Training your assessors, and
2. Supporting your assessors

Just as you train and assess your candidates, you should train and assess your assessors. The basics of valid and reliable assessment of adult trainees are not difficult to understand, nor do they take a great deal of time to master.

Assessors should be chosen from people who are not only expert, professional mariners, but they should also have an interest in training and assessment. The training of the assessors is a topic that deserves an article of its own, but suffice it to say that if the right people are chosen, a two day training session will go a long way to improving your assessment practices.

Once you have chosen and trained your assessors well, you need to support them in their work.

How can you support your assessors? Make sure they have access to documents which clearly lay out how assessment is to be conducted in the organization – both in terms of specific practices and in terms of assessment philosophy. Make sure they have access to the rubrics required to create objective assessment data. And finally, make sure they have a mechanism within the organization where they can easily exchange experiences, suggestions, and questions, regardless of their physical location. By supporting one another, best practices will disseminate quickly and the trainers will be able to act as a group to the benefit of assessment in your organization. Technology can help a lot here through the creation of a company-wide on-line trainer community.

Tip 5: Vary Your Assessments

This may go without saying, but it is very important that your assessments are varied. Having said that, although they should be different in terms of the specific questions which are asked, they should be consistent in terms of the breadth and depth of coverage.

The reason for variation in assessments is that the alternative – always using the same assessment scenarios and questions – will cause candidates to “study to the test”. Eventually the set of questions asked will become common knowledge and the candidates will ensure they know exactly that information.

Some might argue that “studying to the test” is not a bad thing. After all, if they are able to successfully pass the assessment, they are qualified. The problem with that argument is that assessments are never comprehensive. An exhaustive assessment – where all required knowledge and skills are fully assessed, is a practical impossibility.

The best we can do is to “audit” knowledge and skills, similar to how an accountant audits the financial records of a business. We test as much as we can of the most critical concepts, and then we sample the remainder. The aim being that if the candidate performs well on the sampling of knowledge/skills, then we can be reasonably confident that they will perform well on skills not assessed.

However, this only works if the candidate does not know ahead of time specifically what will be tested. The assessment creates the incentive to study all competencies, even those which might not be tested.

So – how do we vary assessments while ensuring that assessments are roughly equal in coverage and difficulty? We do the following:

1. We list the categories of skills and knowledge which we would like each assessment to cover.
2. For each category, we create a set of questions which we feel are roughly of equal importance and equal difficulty.
3. When it comes time to assess, we randomly select a specific number of questions from each category.

This technique can apply for almost any type of assessment – written exams, oral exams, and demonstrative scenarios.

In order for this technique to be successful, the number of questions in each category “question bank” should be significantly larger than the number chosen for each exam. A ratio of at least 10 to 1 (10 questions in the “question bank” to every 1 question chosen for a single assessment) is recommended. This ratio should provide sufficiently randomized assessments that candidates will be unlikely to determine what will be asked.

In the case for multiple choice examinations, technology can be your friend here. For example, many modern LMS’s have implemented ways to deliver randomized, but consistent multiple choice exams.

Tip 6: Implement a Process of Continuous Improvement

Continuous improvement is such an important concept, I do not believe that a discussion of maritime assessments would be complete without mentioning it. What does it mean? Continuous improvement is a process by which we periodically evaluate the performance of a task (in this case, the quality of our assessment practices) with a view toward improving it.

During each continuous improvement cycle, we use Key Performance Indicators (KPIs) which give us a measure of how good our performance currently is. We evaluate the KPIs and then apply some changes that we hope will improve the outcomes. Later, we re-assess the KPIs to determine how the changes we made affected them. If there was a positive effect, we keep the changes. If not, we undo the changes.

In order to constantly be improving our assessment practices, there are a number of KPIs we can consider. For example:

1. We can survey the supervisors in our organization to ask them how successful they believe our assessment practices are in ensuring that unqualified candidates are not put into service.
2. We can ask candidates (both those who were successful and those who were not) whether the assessments accurately reflected the knowledge and skills taught during training. If not – the problem may lie with the assessments or the training, but there is a problem somewhere.
3. We can monitor exam outcomes to look for trends. Are the grades generally increasing? Are they generally decreasing? If so, this may or may not indicate a problem, but it does need to be investigated.
4. We can also monitor the performance of individual questions. Are there questions which are routinely performed poorly or always performed successfully? In the former case, the issue may again be with the training or assessment, but an issue exists. In the latter case, we have a question which is not an effective assessment tool. It may be useful for other reasons (for example, as an incentive to study), but it is telling us nothing about candidate performance

In order to gather these KPIs, technology can, again, be a tremendous help. Without some technological assistance, generating KPI reports is a very time consuming process. Old exams must be gone over question-by-question. Questionnaires must be distributed, collected and assessed. Fortunately, most learning management systems (LMS) are able to support your training, and may be able to generate most of these KPIs automatically.

The bottom line for continuous improvement is that if you are not actively engaged in measuring and improving your performance, then you are passively engaged in performance deterioration. It is just that important.

Conclusion

In the end, assessment is about giving us the data we need in order to make an informed professional decision about individuals. We can only make those decisions if our assessment techniques provide reliable and valid data. If our assessment techniques are flawed or one-dimensional, then the data is incorrect or incomplete and we cannot make an informed decision. It is not difficult to take steps toward more reliable and valid assessments – steps which will improve organizational performance, safety, pride and professionalism as a whole.

Have a wonderful day and thanks for  reading!