How to write valid and reliable multiple-choice questions

All killers, no fillers!

Multiple-choice questions are a fantastic means of capturing evidence of learning.  They are a simple format that most students are familiar with, are easily marked by and assessor or LMS and can provide immediate results.  But to be effective, they need to be valid and reliable.

Validity refers to the question and assessment measuring what it is supposed to measure.  That means the question needs to directly relate to the learning content and the competency outcomes.  There cannot be ambiguity or second guessing. 

Reliability means the question (and assessment) is repeatable, and measures consistent results, over time and between different learners.

We want to design clever questions that measure learning rather than measuring the learner’s ability to take tests!

Simplification of valid and reliable multiple-choice question.

In this example, the question is valid as there is no confusion between the content and the question.   It is also reliable as it should be repeatable across multiple learners and with using ALT text and speech to text, we can ensure it is accessible those with difficulty reading, listening or who are colour blind, still get a fighting chance at answering the question correctly.  Assuming our learners can comprehend English.1

But this is a simple example, it is factual and requires low cognitive burn.  What about complex topics and higher-level learning outcomes.  How can we make questions that are valid and reliable?  Well, it requires a bit of work but rewarding. 

Take the time

Often when designing online (and offline) learning, the question writing comes last in the process and is usually rushed to get the project finished.    If you can, write the questions as you work on each chunk of the learning.   Also write the correct answer down as soon as you have written the questions.  This will give it validity and clarity. 

A competency standard, such as using nationally recognised units of competency provide essential knowledge requirements that should be used to write the questions. 

Writing the stem

The stem is the question, or the problem posed.  The stem could refer to an image, graph or external reference that requires interpretation, or could be a simple question. 

Write the stem as a single, clearly stated problem that is as brief as possible.  Long questions can cause confusion especially for low level learning and add to the time burden.  If possible, use positive rather than negative statements, for example try not to use “which of these is not…?”  Using negative language increases the cognitive load of the learner requiring them to read each answer and decipher which is incorrect.  These can be stumbling block especially coming after a run of positive language questions.

 The question should be meaningful without having to read the answers first and avoid giving clues to the correct response in the stem.

Where possible avoid `fill-in-the-blank’ style multiple-choice questions.  These types of questions __________________.

a) make it easier for the student to find the response in the list of options.  This makes the question less effective.

b) require the student to evaluate the sentence for each alternative.  This can add to the time burden and the cognitive load.

c) require the marker/teacher to analyse each response by reading the question each time.  This adds time to the marking process.

d) make it more difficult for the student as the blank space can distract them.  This distraction can lead to an incorrect response.

The answers is b, but you get the idea.

Writing the response

The learner should be able to select the correct response without having to sort through unnecessary complexities or be able to guess the correct answer based on the way the question and the alternate answers are written.  Writing alternate answers is one of the most difficult aspects of writing valid and reliable multiple-choice questions. 

Ensure alternatives are written to match the grammar, length, and complexity as the correct answer.  The alternatives should be plausible answers and relate to the content which adds to the question’s reliability. 

If possible, avoid using `all of the above’ options as the answer can be derived from partial information.  If the student can see that 2 out of the 4 are correct answers, then `all of the above’ is the correct answer and if they detect one answer is incorrect then `all of the above’ is also incorrect.  The same apples to `none of the above’.  The student is just detecting wrong answers without necessarily knowing what the correct answer is to the question. 

Conclusion

So, as you can see there is a lot to think about when writing multiple-choice questions but by putting the work in, the process of scoring and providing feedback becomes more efficient and immediate.  Validity and reliability should always be at the forefront in the writing process and as with all aspects of online learning and assessment, you need to ensure instructions and questions are as clear as possible to make it fair and equitable