Categories
Assessment and feedback Delivery and assessment of the curriculum Design, development and approval of programmes Digital technologies to enhance learning and teaching and assessment. Enabling student development and achievement qualitative Student engagement in learning Student experience Technology and digital literacies

Experiences of reflection and digital storytelling

Blog Authors: David Cullen, Walter Patterson, John McVeigh, Lynn Brown, Tracey Howe, Lisa Shields

Image: Dave Morris from Oxford, UK / CC BY (https://creativecommons.org/licenses/by/2.0)

City of Glasgow College has many interests including: ESOL teacher development, reflective practice in educators and the use of technology in assessment evidence. This week’s paper ESOL pre-service teachers’ experiences and learning in completing a reflection paper and digital storytelling was chosen for review as it ostensibly covers a number of these topics.

Here’s what they did

The subjects of the qualitative study were 20 students on a post-graduate level Teaching English for Speakers of Other Languages (TESOL) course. In a module on Language and Culture, the students had to complete two assessment tasks: a written assignment and a digital storytelling artefact. The researcher then had the students carry out two further tasks for the purposes of the study: writing a reflection paper and delivering a reflective presentation. The researcher analysed all four sources of data and sought to examine the subjects’ performative approaches to the tasks, and their reactions to the tasks. The researcher also considered the gender and nationality of the subjects in relation to their performance and response.

Here’s what they found

  • that there were general commonalities in the subject’s performance of and reaction to the two sets of tasks (assessment and reflection).
  • there was a significant difference in subjects’ responses between the familiar written report and the unfamiliar digital storytelling task.

The author/s concluded

The inclusion of a dual reflective task was of benefit to pre-service TESOL candidates as it enhanced their reflective literacy and their understanding of the course content on Language and Culture. TESOL training courses should consider using this approach.

Our Journal Club’s views

Who are the authors of the paper and where do they work? This study was undertaken by an individual researcher, an Associate Professor of English (TESOL) at Murray State University in the United States. Regarding the individual author, it was not possible to find a list of publications or citations. We also noted that only one other individual contributed to the research activities.

What do we know about the journal? The Australasian Journal of Educational Technology is a bi-monthly peer-reviewed academic journal covering research in educational technology, instructional design, online and e-learning, educational design, multimedia, computer assisted learning, and related areas. It was rated in 2015 as having an Impact factor of 1.171. It is published by the Australasian Society for Computers in Learning in Tertiary Education. Our view is that this is a trustworthy publication.

What about the methodology used? We felt that the literature review was almost completely descriptive, serving only to provide definitions of terminology, and failed to critically evaluate the sources.

Secondly, the researchers failed to identify and declare any potential bias and limitations of their activity. Thirdly, we felt that the writing of the article, while being thorough and detailed in parts, lacked clarity, and was consequently difficult to decode and interpret.

Finally, there was significant and undeclared potential for bias: the researcher was also the course tutor; the students as subjects were potentially eager to teacher-please in their responses; only one other individual was involved in supporting the researcher and that person was also a direct colleague.

Given the high probability of bias and the other concerns outlined above, we have limited confidence in this article and we feel that the exploratory project would have been better served had it been presented as a less formal case study account.

Our conclusions are – that this evidence has a high risk of bias.

Implications for our practice

How can CoGC staff develop “reflection” in our working practice and professional development ?

Is this concept included in Staff Integration activities?

Is digital storytelling something we can use in staff development and/or student work? The topic of reflective practice should be considered for inclusion in the OneCity event in June and the Education Symposium after summer.

We have recently had a tussle with a local university over the accreditation of a vocational award. The university insisted that reflection should be assessed via an essay. After several rounds of negotiation, the university has accepted that there are equally valid representations of reflective practice – such as digital storytelling (videos, blogs, e-portfolios). We have been assured on many occassions by Scottish Qualifications Agency (SQA) that it has moved away from specifying the form in which evidence can be presented – if only all External Validators were of the same mind!

Next steps

The topic of reflective practice should be considered for inclusion in the OneCity event in June and the Education Symposium after summer.

View from

What do you think?

References

Ho-Ryong Park. ESOL pre-service teachers’ experiences and learning in completing a reflection paper and digital storytelling. Journal of Education Technology, 2019, 35(4)

SQA 2017 Digital Evidence for Internally Assessed HN and VQ Units: Principles and Guidance

Keywords:

Our Blog Posts are written by staff at City of Glasgow College to inform and inspire our practice. We meet together at the Journal Club to consider the latest evidence to provide insights on hot topics related to learning and teaching, quality assurance and subject needs. It forms part of our activity for General Teaching Council Scotland registration and Professional Standards for lecturers in Scotland’s Colleges demonstrating that we are a self-critical staff community.

Categories
Assessment and feedback Delivery and assessment of the curriculum Digital technologies to enhance learning and teaching and assessment. review Student engagement in learning Technology and digital literacies

Student-generated video creation for assessment?

Blog Authors: Fiona Balloch, Jan Robertson, John McVeigh, Robertson, Lisa Shields, Joe Wilson, Tracey Howe

Image: Photo by Hermes Rivera on Unsplash

Student-generated video creation assessments are an innovative and emerging form of assessment in higher education. Academic staff may be reluctant to transform assessment practices without robust evidence of the benefits and rationale for doing so and some guidance regarding how to do so successfully. JISC have recently published Future of assessment: five principles, five targets for 2025 which states ‘In a move away from the traditional essay or exam, assessments are building in authenticity by asking students to develop websites, set up online profiles, shoot and edit videos, and use social media.’

We explored the idea, with reference to the article Student-generated video creation for assessment: can it transform assessment within Higher Education? published in the International Journal of Transformative Research, 2018.

Here’s what they did

They searched literature and conducted a thematic analysis related to the use of student-generated video for assessment.

Here’s what they found

For successful use of video creation for assessment:

  • Align video creation task set to both learning outcomes and skills development required for graduate capabilities for relevant industry
  • Ensure technological support, resources and infrastructure are all in place
  • Have an intentional change management process to support both staff and students in the transition to a new assessment format.
  • Involve students in the generation of clear guidance for the assessment and development of an assessment rubric.

The author/s concluded

Video assessment is beneficial for students’ digital communication skills and an effective and enjoyable method of assessment.

Our Journal Club’s views

Who are the authors of the paper and where do they work? At the time of publication the authors are Ruth Hawley and Cate Allen, who work at University of Derby.  Our view is that the authors may be biased in favour of video assessment, in order to provide evidence to support an initiative taking place within their own institution.

What do we know about the journal? The fully refereed Journal of International Journal of Transformative Research does not seem to be live yet and will be issued for the first time in Fall 2020. Our view is that the journal does not meet its stated aims, as it says that articles should explore transformative impact but this is not the case in this article.

What about the methodology used? Research could not be easily replicated based on the level of detail provided in the paper. In addition, the findings lack critical analysis. that this evidence is inconclusive and biased. It lacks a rationale for the use of video assessment or guidance on how it can be used effectively.

Our conclusions are – that this evidence has a high risk of bias.

Implications for our practice

  • How can Nautical courses extend their use of asynchronous video assessment with international students?
  • How could the COGC Health suite integrate video assessment into assessments?
  • How could issues such as trolling, and confidence with one’s own image on video be addressed through digital communication skills training?
  • How could YouTube and Flipgrid be used for assessment?
  • How issues such as trolling, and confidence with one’s own image on video could be addressed through digital communication skills training. It is easier for assessors to view videos asynchronously at the time of the assessor’s choosing, than assessing a large run of live events, one after the other.
  • Training and support is available through the College Learning and Teaching Academy

Next steps

Create a working group to pursue this topic in the College with a view to group-creation of a paper on this area.

View from

What do you think?

References

Keywords: assessment, video, student-generated, Higher Education, digital, technology

Our Blog Posts are written by staff at City of Glasgow College to inform and inspire our practice. We meet together at the Journal Club to consider the latest evidence to provide insights on hot topics related to learning and teaching, quality assurance and subject needs. It forms part of our activity for General Teaching Council Scotland registration and Professional Standards for lecturers in Scotland’s Colleges demonstrating that we are a self-critical staff community.

Categories
Assessment and feedback Delivery and assessment of the curriculum Digital technologies to enhance learning and teaching and assessment. qualitative quantitative Technology and digital literacies

Rubrics in Vocational Education

Blog Authors: Tracey Howe, John McVeigh, David Cullen, Walter Patterson, Ian Hamilton

Image by Cleonard1973 / CC BY-SA (https://creativecommons.org/licenses/by-sa/4.0)

Our College delivers vocational training that frequently uses observation-based assessment. However we realise that for this to be reliable, fair, and practicable it also needs to demonstrate consistency across assessors (quality assurance), and involve decisions about the range and number of observations of performance that are required to make a reliable judgement about competence. The notion of using rubrics is being explored and we looked at this paper ‘Electronic Rubrics Design to Assess Student Competence in Vocational Education‘.

Here’s what they did. Using Design Based Research they aimed to develop an instrument that contained a rubric on food and beverage service practice in vocational education that is valid, practical, and effective. The three stages included: 1) identification and analysis of problems, 2) development of prototype program, 3) test and prototype implementation of the program.

They explored the needs of 4 lecturers from food and beverage service of different universities and 30 students of culinary education Indonesian Education University. This defined the concept of evaluation tools that were made and validity was explored using the view of 2 specialist subject matter experts and 1 assessment expert.

Data collection involved interviews and questionnaires and descriptive statistics.

Here’s what they found.

  • food and beverage service lecturers have never created nor applied an assessment rubric.
  • students on food and beverage service programme do not know the assessment tools used by lecturers
  • researchers designed a task performance guide that can be used by students in the practical implementation.
  • the performance criteria for the task and performance assessment (rubric) showed a good degree of validation

The author/s concluded

The results of the study consisted of instruments used in food and beverage service performance task of student assignments as a guide for students in carrying out lab work and performance assessment consisting of electronics rubric as practical competency guidelines. The results of the development were validated, based on expert discussions conducted using the Aiken index coefficient.

Our Journal Club’s views

Who are the authors of the paper and where do they work? All authors work at UNIVERSITAS PENDIDIKAN INDONESIA,UPI The Education University.

What do we know about the journal? This paper was published as part of proceedings from the 1st Vocational Education International Conference (VEIC 2019).

What about the methodology used? The main problem with the paper was that clearly English was not the authors’ first language. This resulted in lack of clarity and understanding throughout. The methodology was unclear and all subsequent analysis, results and conclusions were difficult to interpret.

Our conclusions are – that this evidence has a high risk of bias.

Implications for our practice

There are a number of individuals and programme teams across our College developing and using electronic rubrics. These include Beauty and Culinary Arts and are used on ‘Moodle’ our VLE platform. It was felt that rubrics give a more standardised feedback to student that allows their understanding of their performance.

A key area where we could look at this is that of ‘meta skills’ as these are cross disciplinary in nature and could provide core methodology and consistency of approach.

Next steps

  • College staff currently developing or using rubrics could showcase their work at forthcoming internal events and conferences.
  • We could propose a work package on rubrics as part of the current institutional review of assessment and feedback
  • Create a working group of interested individuals
  • Ask OD and COPTE for staff development in this area
  • Look at the Skills Development Scotland meta skills

View from

What do you think?

References

  • Muktiarni, M. et al. (2019) ‘Electronic Rubrics Design to Assess Student Competence in Vocational Education’, in 1st Vocational Education International Conference (VEIC 2019). Atlantis Press, pp. 257–261. doi: 10.2991/assehr.k.191217.042.

Keywords: rubrics, assessment, competence, vocational, college

Our Blog Posts are written by staff at City of Glasgow College to inform and inspire our practice. We meet together at the Journal Club to consider the latest evidence to provide insights on hot topics related to learning and teaching, quality assurance and subject needs. It forms part of our activity for General Teaching Council Scotland registration and Professional Standards for lecturers in Scotland’s Colleges demonstrating that we are a self-critical staff community.

css.php