Introduction

Multiple choice tests are one of the oldest assessment techniques in existence. Yet they are also one of the most highly maligned. Why, then, are they so common in maritime training? Are they effective? The answers to these questions are not as simple as one might hope. However, every maritime trainer needs to be aware of the strengths and weaknesses of multiple choice question (MCQs), and how to use them to greatest effect.

This article is the first of a series that looks at MCQs in maritime training. This first post examines the strengths and weaknesses of MCQs. Subsequent posts will offer some tips on how to create and use MCQs to take advantage of their strengths.

Criticisms of Multiple Choice Questions

Multiple choice question (MCQ) exams have been used for over 100 years to assess training effectiveness both within and outside the maritime industry. Even so, trainers and educational researchers love to hate them. There are many valid arguments frequently levied against MCQs:

One of the main arguments is that MCQs can’t test higher order thinking, but instead only test factual knowledge.

This is not technically correct; MCQs can be constructed to test higher order thinking, but it requires skill to construct such questions. And even in the case that such questions are constructed well, they are unlikely to be as effective at the assessment of reasoning ability as are other forms of assessment like written responses, oral responses, or demonstrations of skills.

Performance on an MCQ can generally only fit into one of two categories: correct or incorrect.

The problem is that a “black and white” analysis does not reflect the reality that knowledge and competencies are rarely either completely known or completely unknown. Candidates are much more likely to have a partial grasp of the knowledge being measured.

As before, MCQs can be constructed to test partial knowledge, but doing so requires some skill. And even if that effort is put in, they are unlikely to be as effective as other forms of assessment at revealing partial knowledge.

MCQs are frequently written ambiguously, are confusing, and can sometimes be guessed correctly.

These are all examples of a general class of problem with MCQs: simply said, they are hard to write well.

I can personally attest to this fact. At British Columbia Ferry Services Inc (BC Ferries) there is a testbank of over 13,000 questions which have largely been written by BC Ferries subject matter experts. Despite the high level of expertise and the great deal of effort put into the creation of these questions, it is not unusual at all for questions to require multiple cycles of use and review before they are considered to be well written and unambiguous. Constructing well-written MCQs is not as easy as one might imagine.

Why are MCQs Popular?

This is certainly not an exhaustive list of criticism of MCQs. There are others. With all of these drawbacks, one might wonder why they remain so popular.

One of the obvious reasons is that they are easy to grade. And as a former faculty member myself, I can tell you that I and most of my colleagues considered grading exams only slightly preferable to having one’s eyes gouged out with blunt sticks. This adds fuel to the fire of arguments against their use: not only do MCQs have many shortcomings, but their main reason for use is arguably because trainers are too lazy to grade answers to “better” assessment methods.

However, it is important to consider some of the benefits of MCQs. They actually have many, and these benefits need to be considered carefully. MCQs bring to the table benefits that other assessment mechanisms cannot offer. As one component of a comprehensive assessment strategy, MCQs are a valuable tool. However, their value can only be realized if they are used for what it is best at.

So – what are MCQs best at?

The Strengths of Multiple Choice Questions

They can Assess Factual Knowledge

One advantage of MCQs is that they are quite effective at testing well defined factual knowledge. This is important in the maritime industry because all skills and competencies are rooted in a foundation of knowledge.

While knowledge itself is not sufficient to ensure that a competency is held, it is necessary. That is, even if a trainee can demonstrate a skill under assessment conditions, if they do not possess the knowledge underlying that skill, then they cannot be expected to perform the skill safely under varying conditions. Knowledge must be tested separately as part of any comprehensive assessment strategy, and MCQs can be very effective at testing certain forms of knowledge.

They are Objective

Another significant advantage of MCQs is that they completely remove subjectivity in grading. In my view this is one of their greatest strengths. Variability in grading is a pervasive problem for any large organization. Regardless of how standardized the training may be, different trainers will have different ideas of what constitutes a “pass” or “fail”. This problem can be somewhat alleviated through the use of rubrics, but even these can never completely remove the subjectivity inherent in other assessment techniques. The grading of MCQs is 100% objective – a valuable contribution to an assessment program.

They Enable Frequent Formative and Summative Assessment

One of the main reasons for the popularity of MCQs, as mentioned above, is that they are easy to grade. This is especially true if they are done online where grading is automatic. This is obviously an advantage to the trainer. However, it is, in fact, an even more important advantage for the trainee.

The reason for this is that frequent and regular assessment is a cornerstone of effective training. Assessment comes in two flavors – formative and summative. Summative assessment is used at the end of training to determine whether someone has learned the concepts and therefore is fit for duty. It is, in general, for the benefit of the training organization.

Formative assessment is different. It exists to benefit the trainee and inform the training process. It can be used throughout training to provide feedback to the trainee as to how effectively the lessons are being learned. The more frequent the feedback through formative assessment, the earlier (and less costly) the training course correction are. The problem, of course, is that most forms of assessment are costly. Therefore, although formative assessment is critically important, it is rarely provided in sufficient quantity.

This is where MCQs are especially useful. MCQs, especially those delivered online, require no time to grade. They can be applied as frequently as desired as long as there are sufficient questions available. In the absence of an instructor who has sufficient free time to grade frequent formative assessment, MCQs are an exceedingly valuable tool as they enable frequent and regular formative assessment.

They are Easy to Randomize

Another feature of MCQ tests is that technology can be used to randomize the exams such that no two exams will be exactly the same. For those not familiar with MCQ question randomization, most learning management systems (LMS) have the ability to dynamically create exams by randomly selecting questions from an exam database. In the case of our LMS at Marine Learning Systems, we employ a form of intelligent randomization where the difficulty and coverage of questions on every exam is normalized. This helps ensure that all candidates receive exams of the same coverage and relatively equal difficulty even though the question mix is different.

The ability to randomize exams has two advantages over other kinds of assessment methodologies.

First, randomized examinations further enhance the opportunities for frequent and regular formative assessments. Randomized examinations allow trainees to continue to assess themselves, as each time they do so, they can receive a somewhat different examination.

Second, for summative assessments, randomized exams ensure that no two trainees receive the same exam. This helps prevent the opportunity for information about the exam to spread from one trainee to another.

They are Transparent: Analytics

Another wonderful advantage of technology-delivered MCQs is the ability for them to generate analytics which give training managers excellent visibility into the learning process.

Each time a trainee answers a MCQ online, a record is made of that response. This data, when analysed and presented, can provide insights into how well trainees – both as individuals and collectively – are learning. For example, which knowledge and concepts are not being learned well? What common misconceptions do our trainees have?

These and many more questions can be answered by MCQ analytics, yet they can be difficult at best (and impossible at worst) to obtain from most other types of assessment. If the goal is to identify gaps in knowledge early – before they lead to performance or safety issues, MCQ analytics can help tremendously.

Conclusion

So – are MCQs of value despite their obvious shortcomings? I say yes – absolutely.

Should they be used as the sole technique to assess seafarers? Absolutely not.

In fact, there is no single technique which could be used as the sole technique for assessment. Each assessment tool offers specific advantages. And while each tool has limitations, those limitations are accommodated by the strengths of the other tools. MCQs are no exception. Therefore, by combining assessment techniques in your overall assessment strategy, you can take advantage of the benefits of each.

The next article in this series is going to present some guidelines on how to create good multiple choice questions. While it is not difficult to do so, a little bit of advice goes a long way in ensuring the time you put into it yields good results.

If you would like to receive notification of the next article of this series (and subsequent maritime training articles) and have not yet done so, please subscribe to the blog below to to receive notifications.

Until then – thanks for reading!