Blog Authors: Tracey Howe, John McVeigh, David Cullen, Walter Patterson, Ian Hamilton
Image by Cleonard1973 / CC BY-SA (https://creativecommons.org/licenses/by-sa/4.0)
Our College delivers vocational training that frequently uses observation-based assessment. However we realise that for this to be reliable, fair, and practicable it also needs to demonstrate consistency across assessors (quality assurance), and involve decisions about the range and number of observations of performance that are required to make a reliable judgement about competence. The notion of using rubrics is being explored and we looked at this paper ‘Electronic Rubrics Design to Assess Student Competence in Vocational Education‘.
Here’s what they did. Using Design Based Research they aimed to develop an instrument that contained a rubric on food and beverage service practice in vocational education that is valid, practical, and effective. The three stages included: 1) identification and analysis of problems, 2) development of prototype program, 3) test and prototype implementation of the program.
They explored the needs of 4 lecturers from food and beverage service of different universities and 30 students of culinary education Indonesian Education University. This defined the concept of evaluation tools that were made and validity was explored using the view of 2 specialist subject matter experts and 1 assessment expert.
Data collection involved interviews and questionnaires and descriptive statistics.
Here’s what they found.
- food and beverage service lecturers have never created nor applied an assessment rubric.
- students on food and beverage service programme do not know the assessment tools used by lecturers
- researchers designed a task performance guide that can be used by students in the practical implementation.
- the performance criteria for the task and performance assessment (rubric) showed a good degree of validation
The author/s concluded
The results of the study consisted of instruments used in food and beverage service performance task of student assignments as a guide for students in carrying out lab work and performance assessment consisting of electronics rubric as practical competency guidelines. The results of the development were validated, based on expert discussions conducted using the Aiken index coefficient.
Our Journal Club’s views
Who are the authors of the paper and where do they work? All authors work at UNIVERSITAS PENDIDIKAN INDONESIA,UPI The Education University.
What do we know about the journal? This paper was published as part of proceedings from the 1st Vocational Education International Conference (VEIC 2019).
What about the methodology used? The main problem with the paper was that clearly English was not the authors’ first language. This resulted in lack of clarity and understanding throughout. The methodology was unclear and all subsequent analysis, results and conclusions were difficult to interpret.
Our conclusions are – that this evidence has a high risk of bias.
Implications for our practice
There are a number of individuals and programme teams across our College developing and using electronic rubrics. These include Beauty and Culinary Arts and are used on ‘Moodle’ our VLE platform. It was felt that rubrics give a more standardised feedback to student that allows their understanding of their performance.
A key area where we could look at this is that of ‘meta skills’ as these are cross disciplinary in nature and could provide core methodology and consistency of approach.
Next steps
- College staff currently developing or using rubrics could showcase their work at forthcoming internal events and conferences.
- We could propose a work package on rubrics as part of the current institutional review of assessment and feedback
- Create a working group of interested individuals
- Ask OD and COPTE for staff development in this area
- Look at the Skills Development Scotland meta skills
View from
What do you think?
References
- Muktiarni, M. et al. (2019) ‘Electronic Rubrics Design to Assess Student Competence in Vocational Education’, in 1st Vocational Education International Conference (VEIC 2019). Atlantis Press, pp. 257–261. doi: 10.2991/assehr.k.191217.042.
Keywords: rubrics, assessment, competence, vocational, college
Our Blog Posts are written by staff at City of Glasgow College to inform and inspire our practice. We meet together at the Journal Club to consider the latest evidence to provide insights on hot topics related to learning and teaching, quality assurance and subject needs. It forms part of our activity for General Teaching Council Scotland registration and Professional Standards for lecturers in Scotland’s Colleges demonstrating that we are a self-critical staff community.
