Month: February 2020

Categories
Assessment and feedback Delivery and assessment of the curriculum Digital technologies to enhance learning and teaching and assessment. qualitative quantitative Technology and digital literacies

Rubrics in Vocational Education

cartoon of rubric

Blog Authors: Tracey Howe, John McVeigh, David Cullen, Walter Patterson, Ian Hamilton

Image by Cleonard1973 / CC BY-SA (https://creativecommons.org/licenses/by-sa/4.0)

Our College delivers vocational training that frequently uses observation-based assessment. However we realise that for this to be reliable, fair, and practicable it also needs to demonstrate consistency across assessors (quality assurance), and involve decisions about the range and number of observations of performance that are required to make a reliable judgement about competence. The notion of using rubrics is being explored and we looked at this paper ‘Electronic Rubrics Design to Assess Student Competence in Vocational Education‘.

Here’s what they did. Using Design Based Research they aimed to develop an instrument that contained a rubric on food and beverage service practice in vocational education that is valid, practical, and effective. The three stages included: 1) identification and analysis of problems, 2) development of prototype program, 3) test and prototype implementation of the program.

They explored the needs of 4 lecturers from food and beverage service of different universities and 30 students of culinary education Indonesian Education University. This defined the concept of evaluation tools that were made and validity was explored using the view of 2 specialist subject matter experts and 1 assessment expert.

Data collection involved interviews and questionnaires and descriptive statistics.

Here’s what they found.

  • food and beverage service lecturers have never created nor applied an assessment rubric.
  • students on food and beverage service programme do not know the assessment tools used by lecturers
  • researchers designed a task performance guide that can be used by students in the practical implementation.
  • the performance criteria for the task and performance assessment (rubric) showed a good degree of validation

The author/s concluded

The results of the study consisted of instruments used in food and beverage service performance task of student assignments as a guide for students in carrying out lab work and performance assessment consisting of electronics rubric as practical competency guidelines. The results of the development were validated, based on expert discussions conducted using the Aiken index coefficient.

Our Journal Club’s views

Who are the authors of the paper and where do they work? All authors work at UNIVERSITAS PENDIDIKAN INDONESIA,UPI The Education University.

What do we know about the journal? This paper was published as part of proceedings from the 1st Vocational Education International Conference (VEIC 2019).

What about the methodology used? The main problem with the paper was that clearly English was not the authors’ first language. This resulted in lack of clarity and understanding throughout. The methodology was unclear and all subsequent analysis, results and conclusions were difficult to interpret.

Our conclusions are – that this evidence has a high risk of bias.

Implications for our practice

There are a number of individuals and programme teams across our College developing and using electronic rubrics. These include Beauty and Culinary Arts and are used on ‘Moodle’ our VLE platform. It was felt that rubrics give a more standardised feedback to student that allows their understanding of their performance.

A key area where we could look at this is that of ‘meta skills’ as these are cross disciplinary in nature and could provide core methodology and consistency of approach.

Next steps

  • College staff currently developing or using rubrics could showcase their work at forthcoming internal events and conferences.
  • We could propose a work package on rubrics as part of the current institutional review of assessment and feedback
  • Create a working group of interested individuals
  • Ask OD and COPTE for staff development in this area
  • Look at the Skills Development Scotland meta skills

View from

What do you think?

References

  • Muktiarni, M. et al. (2019) ‘Electronic Rubrics Design to Assess Student Competence in Vocational Education’, in 1st Vocational Education International Conference (VEIC 2019). Atlantis Press, pp. 257–261. doi: 10.2991/assehr.k.191217.042.

Keywords: rubrics, assessment, competence, vocational, college

Our Blog Posts are written by staff at City of Glasgow College to inform and inspire our practice. We meet together at the Journal Club to consider the latest evidence to provide insights on hot topics related to learning and teaching, quality assurance and subject needs. It forms part of our activity for General Teaching Council Scotland registration and Professional Standards for lecturers in Scotland’s Colleges demonstrating that we are a self-critical staff community.

Categories
Quality assurance review

What is quality assurance and why do we need it?

Blog Authors: Tony Adams, Fiona Balloch, Lynn Brown, Tom Duff, Tracey Howe, John McVeigh, Joe Mulholland, Penny Robertson, Lisa Shields, Derek Timpany, Joe Wilson.

There have been a number of recent conversations across our College about Quality Assurance (QA). What is it? Why do we need it? To help us consider these questions and aid our conversations going forward we took a look at the literature.  

In 2015 Tricia Ryan from the Laureate International Universities network published a review of the literature of quality assurance in higher education in Higher Learning Research Communications an open access journal. 

Here’s what they did. The author reviewed a series of papers relating to quality assurance in global higher education. It provides an overview of accreditation as a mechanism to ensure quality in higher education, examines models of Quality Assurance, and explores the concept of quality (including definitions of quality and quality assurance).  In addition, this paper provides a review of research on the effectiveness of quality assurance practices, with a particular focus on student involvement with quality assurance. 

Here’s what they found

  • there is no agreement as to a QA definition or a QA model. 
  • accreditation structures are decentralized and complex at both the regional and international level. 
  • concerns of faculty members and other stakeholders, such as students, about the QA process 

The author/s concluded

  • There is a need for a common framework for a quality assurance model; however, there is no agreement as to a QA definition or a QA model.  
  • Given that students are at the centre of higher education, and invest time and money in the system, involving them could improve QA processes 

Our Journal Club’s views

Who are the authors of the paper and where do they work? At the time of publication Tricia Ryan worked, Director, Quality and Performance Improvement, Academic Quality and Accreditation, The Laureate International Universities network. This network comprises over 25 higher education institutions and more than 875,000 students, primarily focused in Latin America. Our view is that this author has validity. 

What do we know about the journal? The Journal Higher Learning Research Communications is a peer reviewed journal founded in 2011 by four Laureate International Universities® network institutions: Istanbul Bilgi University (Turkey), Universidad Andrés Bello (Chile), Universidad Europea de Madrid (Spain), and Walden University (USA). It is an open access journal. Our view is that this is a reputable journal. 

What about the methodology used? This is a narrative literature review. The aim of the review is stated as “to examine the literature surrounding quality assurance in global higher education. However, the author does not state any specific question/s. There are no details on why or how the included papers were selected. Nor are there any details on whether any quality appraisal of each paper was undertaken. There is only one author and so only one opinion. Although the aim was to consider global higher education the majority of papers focus on the US. All included papers included appeared to be in English Language. The included papers are however relatively recent compared to the publication date. 

Our conclusions are – that this evidence has a potential high risk of bias.

Implications for our practice

Do we have a shared vision for quality and if so, what is it?  Quality is quite a nebulous concept, one that almost defies analysis and definition. One instinctively knows or feels if one has experienced a quality product or service but putting the finger on what exactly made that an experience of quality can be difficult to evaluate….do we need to? 

Within education there are different stakeholders who wish to see that quality is attained. From a macro level, one could view this from society itself, as within educational institutions the student is exposed to the values, mores and norms of that particular nation state, so this would by default be a component of the quality of the education.  The providers of that education wish to deliver a service of quality, both in terms of the process and of course the outcome, in terms of results. This is likely to be determined and influenced by an external body and financial sanctions, with the expectation and demand of ever-increasing numbers of students successfully achieving, and at higher levels. This expectation in turn could negatively impact of the overall quality of the service provided, the value of the qualification obtained and the reputation of the awarding institution. At the ‘coal face’, teachers or lecturers teach for many reasons, but one would suggest that they all wish to be ‘good’ teachers, inspirational and motivating, not only from their own perspective but most certainly in their students’ eyes, so will strive to provide ’quality’ teaching…. but how is this defined? 

Ultimately, it could be suggested that is in the student, the person being taught who can determine the quality of the teaching and the educational experience as a whole. We can all think of exceptional teachers in our lives and can recognise quality teaching…so it may be, that this group is the one who trumps the others in terms of determining quality…and perhaps this is where we should go in order to try to establish a working definition of the concept.  It may be that what is uncovered by such an approach will differ or reflect the concept of quality education from the other relevant stakeholders. Perhaps asking the students might help to establish a baseline to define what is high quality teaching and education…. if we feel the need to capture its essence. 

What internal mechanisms do we have in place for Quality Assurance? This includes but is not limited to;  

  • In course questionnaires such as student satisfaction, learning and teaching questionnaire 
  • Students representation – Board of Management, faculty boards, CIAMS, academic board and student academic experience committee, student services committee, learning and teaching committee 
  • Guidance sessions  
  • Myvoice.  

We think in terms of the paper City of Glasgow College are probably ahead of HE globally, however, the paper was potential for high bias and was published in 2015. 

What external mechanisms do we have in place for Quality Assurance?  We have SQA External Verification, Education Scotland, University partners, Professional bodies who; 

  • Talk to staff and students 
  • Verify assessments 
  • Award qualifications 

In terms of the material discussed in the paper we are perhaps ahead of international model.   

Our challenge is the feedback loop – do we take action on what learners say and what we are analysing the projects. We have different models in College dependent on the mode of student. 

Other topics We them moved into wider ranging conversation about what constitutes quality of learner experience and how we add value and attract learners in an increasingly competitive market place.  How should we react to Amazon Colleges etc and new competition?  What gives us a good reputation? is it quality, how do we define this?  How do we add value? 

  • Education is a social experience, how can we engage learners in fundraising, social responsibility, social learning, broader experiences  
  • Education is not just about getting a certificate  
  • Why did you choose us … we need to do more …where is blended learning and flexible offers.  

How do we quality assure online and blended learning? This was thought to be important and something we will consider at our next meeting. 

Next steps

We agreed that as a group we would draft a paper on College quality systems

View from

City of Glasgow College’s Quality Management Systems   

Information relating to quality can be found on Connected on the Quality Management Systems section. Other information includes; 

References

Keywords: Quality, assurance, verification, education 

What do you think?

Our Blog Posts are written by staff at City of Glasgow College to inform and inspire our practice. We meet together at the Journal Club to consider the latest evidence to provide insights on hot topics related to learning and teaching, quality assurance and subject needs. It forms part of our activity for General Teaching Council Scotland registration and Professional Standards for lecturers in Scotland’s Colleges demonstrating that we are a self-critical staff community.

css.php