Photo above taken by US Navy Mass Communication Specialist 3rd Class Anna Van Nuys / CC BY 2.0

The following post is the first in a series of articles originally written by Murray Goldberg for Maritime Professional (found here). We have decided to re-post the entire series based on reader feedback.

Introduction

This is the first in a series of articles discussing current and best-practice assessment methods in maritime job training and familiarization. Specifically, I am speaking about the testing vessel operators administer to crew to determine whether they are sufficiently prepared to perform their duties on their assigned vessels safely and efficiently.

In planning this series of articles, I have reminded myself how much there is to say about assessment. It is a complex and rich topic and I expect to generate a number of articles on the subject. It is my hope that these articles will provide a basic understanding of assessment principles – an understanding that everyone involved in maritime training should have.

This short introductory article discusses the limits and purpose of assessment, creating a foundation for subsequent articles on assessment. Subsequent articles will look at assessment reliability and validity, professional judgement, the goals and topics of assessments, and the merits of specific assessment practices in the maritime industry. Please follow this blog to receive notification of those upcoming articles.

Assessment is Primary

It is often the case that we give a great deal more thought to training than we do to assessment. This is unfortunate because training cannot be successful (or at the very least cannot be shown to be successful) without an objective and comprehensive assessment process. Your training may be excellent at this moment, but without quality assessments you have no way of knowing this for sure, and you won’t have the tools necessary to keep it on track and continuously improve it. We need to realize that assessment is a critical and necessary part of training, not just something we do at the end in order to apply a credential. It is a primary safety and operations tool to:

1. Determine whether a candidate is fit for duty
2. Determine what gaps in knowledge and skills exist for a candidate
3. Provide key performance indicators for your organization to be used as a basis for analysis and continuous improvement.

Does Anyone Really Understand Assessment?

Assessment is an activity that very few organizations do well, and fewer still understand well.

Assessment is Hard

There is good reason for that. It is not a cut and dried science. Assessment is often based on intuition rather than concrete fact. Assessment tries to peer into the future of an individual and answer the questions: “does he or she have the knowledge necessary to perform when called upon” and “can he or she perform this skill”?

But how can we truthfully say? After all, so much of what we would like to assess is hidden from view, inside the head of the candidate. And while the candidate may be able to demonstrate a skill under one set of conditions, what if those conditions change?

Actually, Maritime Assessment is Even Harder

But as hard as it is to assess someone’s skills and knowledge, true assessment in the maritime industry needs to do more than that. It needs to assess their cognitive abilities as well. Can the candidate assimilate contrasting information and synthesize it into a plan of action when presented with unexpected events or incidents? There is simply no way to know for sure. As such, it is something that the academic community has been wrestling with for ages, and many consider it to be as much art as it is science.

Having said this, there is still much we can do to improve the validity and reliability of the assessments we administer on board a vessel. A little bit of knowledge and planning can go a long way.

The Purpose of Assessment

One of the first things we must realize when designing an assessment program for the maritime is that, as I alluded to in the paragraphs above, full assessment is an impossibility. You cannot devise an assessment program which will completely assess a candidate’s knowledge or abilities.

Instead, at best, assessment is a statistical process – much like an audit, that samples bits of knowledge here or components of an ability there, and assigns a score which is an extrapolation of the sample taken. If the sample size is very small or the assessment techniques are flawed (or both, as is sometimes the case), then the margin of error is going to be very large. This causes the assessment to be inaccurate much of the time.

However, even with a reasonable “sample size” and sound techniques, assessments can never be treated as absolute indicators. Some candidates will assess well and perform poorly, while others will assess poorly and perform well. This begs the question “If assessment is flawed, then why do we assess”?

Having been a university faculty member for 10 years (and one who really dislikes grading exams) I have often asked myself that question. But it turns out there are very good answers. I will list two of them.

Assessment Provide Data to Inform Decisions

First is the obvious answer. In the absence of any other indicators about a candidate, an imperfect assessment is usually better than no assessment at all.

Some form of assessment is required to obtain an estimate of gaps in knowledge and abilities as well as the prospects for future performance. Since any one assessment is imperfect, we should not treat its results as an absolute indicator of knowledge or competence. Having said that, even an imperfect assessment provides data that, when combined with professional judgement, can be used to make decisions.

Assessment is Incentive to Learn

The second answer is, to me, the most important. If nothing else, assessment is incentive to learn. Every candidate knows that successful assessment performance is their key to employment. They are also keenly aware that not everything they need to know or do will be tested. But in the absence of knowing what will be specifically tested, they are faced with having to learn as much as they possibly can about all testable knowledge and skills. There is no greater incentive to deep and broad learning than an assessment.

This is an important fact to keep in mind because anything a trainer does to purposely, or inadvertently, “teach to the test”, or make candidates aware of the specific knowledge their assessment will cover, takes away their incentive to learn as much as they can. The implications of this statement for assessment techniques will be discussed further in subsequent articles on the subject.

Conclusion

We need to always keep in mind that assessment is a largely imperfect exercise. Knowing this should cause us to place assessment results in perspective, to ensure our assessment techniques create an incentive to learn, and to take a keen interest in other indicators of our mariners’ abilities. Most importantly, we should treat assessment as a tool which informs conclusions, not a conclusion in itself.

Subsequent assessment articles will look at how we assess, what we assess, and specific techniques to make our assessments reliable and valid. Please “Follow this blog”, below, to be informed of these articles as they are released.