How to develop assessment criteria in Competency-based Assessment

We recently ran a Professional Development session for our community of graduates and VET practitioners on the topic of Developing Assessment Criteria.

I wanted to explore this concept in detail as it is often confused, and some Certificate IV in Training and Assessment courses do not cover it adequately, leaving people guessing, or adopting poor practice.

Slide show and webinar recording

This article comes from our recent webinar. The slides are below, along with an accompanying article.

WATCH RECORDING

What are the benefits of well-written assessment criteria?

Having well-written assessment criteria creates a common understanding between students, teachers, assessors, managers, employers, regulators, and others. It helps to give transparency around the assessment process, so everyone’s on the same page. It sets clear expectations and helps eliminate confusion.

Further, it allows for accurate assessment records that are defensible. If your criteria are clear and specific, you can confidently conduct an assessment and the student can more confidently participate. In the event that a student is found to be not-yet-competent, you can very clearly articulate why, and point out which criteria weren’t met and why.

It also helps meet the principles of fairness, reliability and builds confidence in the process.

What are assessment criteria used for?

We can use assessment criteria to measure:

• A person’s performance and attitudes
• The qualities and characteristics of a product of someone’s work
• Someone else’s perspective of a person’s performance or product (third-party report)
• Knowledge

We typically have two broad objectives in using criteria in education:

1. Formative assessment – to check progress towards outcomes
2. Summative assessment – to check to see if outcomes have been achieved

Confusing Criteria for Performance vs Product

Assessment criteria can be used to:

1. measure the quality of the performance of someone doing something (observable behaviours)
2. measure the quality of the result of someone doing something (product characteristics)

One of the biggest issues is when these two are confused, or mixed together.

As an assessor, if you are administering an observation assessment, but your observation checklist includes a number of product characteristics, you might find it very challenging to gather evidence that meets many items on your checklist. Similarly, the student may be confused and anxious because they will also be wondering how they can demonstrate those things.

Conversely, and perhaps even more troubling, is if you are an assessor who is trying to determine the quality of someone’s performance by looking solely at the product of their work, you may find yourself guessing about a the student’s performance.

Look at the following examples from various units of competency, and notice how they are distinctly performance-based or product-based.

Of course, it’s not always this simple and there are multiple things that need to be ‘unpacked’ from the criteria that will require both live observation and a review of the result.

Can I simply copy/paste performance criteria from a unit of competency into an assessment tool?

Many people are tempted to do this. Indeed, some units of competency may be written so specifically that this may be possible, although this is rare.

Here’s an example of a unit of competency which has some performance criteria which may be suitable to use almost directly (though bear in mind I am not an industry expert in concreting!)  This is merely to illustrate how specific the criteria would need to be in order to warrant copy-pasting the performance criteria:

Usually, a single performance criterion in a unit of competency have tied up within them a combination of things you that you would need to:

• see being performed live as part of an observation
• inspect the quality of the product created as a result of the performance
• ask questions about

For many units, using the performance criteria wording verbatim will paint with too broad brush strokes to enable a fair and reliable assessment.

Let’s explore some examples.

In these examples we look at how we might ‘unpack’ a performance criterion to create some sort of assessment criteria.

These are not intended to be perfect examples of criteria, but are more to illustrate the process of unpacking the unit.

Use adverbs and adjectives with care

We need to be careful in how we use adverbs (words that modify adjectives and verbs) and adjectives:

• Surface is washed  effectively 
• Candidate  did well  when operating the chainsaw
• Area for painting is prepared  thoroughly 
• Maneuver is performed  very quickly 

This introduce a problematic level of ambiguity and subjectivity into the assessment and cause issues with validity and reliability. We all have different expectations, interpretations and experiences that will impact the assessment decision we make if these vague terms are used too much.

General principles to follow in developing assessment criteria

1. Be as specific and measurable in your wording as possible
2. Be aware of the differences between:
   • Assessing performance
   • Assessing attitudes
   • Assessing product qualities
   • Assessing knowledge
3. Take time to break up the performance criteria into specific and measurable components

Is it possible to remove all ambiguity and subjectivity from assessment criteria?

The more complex the unit of competency, the more challenging this becomes.

Dr. Clair Hughes’ “Quickbite: Practical Guidelines for Writing Assessment Criteria and Standards” (University of Queensland, Australia) was written more with higher-education in mind, however we can take some comforting words from this publication into our work as Vocational Education and Training practioners.

Dr. Hughes writes that assessment checklists can rarely carry “all the detail of the explicit and implicit understandings and skills.”

Any efforts made to reduce ambiguity will be worthwhile in moving towards greater validity and reliability in your assessment tools.

ASQA’s Guide to Assessment Tools is a good resource which talks in detail about the importance of creating tools that allow assessors to use their judgement to determine if someone is competent or not.

Here are some excerpts of note:

• “criteria used to  judge  the quality of performance”
• “also referred to as the ‘assessment decision-making rules’”
• “the rules used to make  judgements  about whether competency has been achieved.”
• “used by an assessor to ensure consistent outcomes when checking evidence quality”
• “should not be open to interpretation.”
• “Must provide sufficient clarity for assessment  judgements  to be consistent across a range of assessors and points in time.”

We can see the term ‘judgement’ features prominently throughout the guide. What does judgement mean?

“the ability to make considered decisions or come to sensible conclusions.”

Good or poor examples?

So now you are aware of the characteristics of good assessment criteria, what do you think of the following examples, good or poor?

• Adjust stem fasteners to 5 nm using a torque wrench
• Prepare baked goods to be ready for customers using a range of techniques and tools correctly and efficiently and in way that reflects a real life commercial baking setting using ingredients on hand.
• Dig hole to the correct depth
• Conduct pre-start checks of excavator in accordance with the Standard Operating Procedures.