Theme: Assessment across the continuum/across borders

Download a PDF of the abstracts: Assessment across the continuum/across borders

Welsh translation of Year 1 single-best answer examinations

  • Professor Phil Smith, Cardiff University School of Medicine
  • Corresponding authors *Smith PE, Whittam S, Tayyaba S.

Background

The Welsh and English languages have equal legal status in Wales; Cardiff University offers all students the option of Welsh translated assessments, regardless of language of tuition. In practice, students and staff are unclear about available entitlements. Previous data show Year 1 students from Welsh medium educational background experience specific difficulties with single-best answer (SBA) questions, deliberately framed around clinical uncertainty and subtlety. We therefore have a duty to support students transitioning from Welsh medium education.

Methods

In 2016, we offered a Welsh translation of Year 1 written summative papers, with the University rather than the medical school meeting translation costs. Year 1 in Cardiff has summative assessments in knowledge only, with no summative practical or clinical examination. The written assessment comprises two papers each of 120 SBAs (~7500 words) with a re-sit opportunity (30 000 words for translation). Following international psychometric translation best practice, the papers were finalised early, translated into Welsh, and then back-translated into English. Welsh-speaking medical personnel compared the back-translated (English) version to the original English paper, changing as appropriate. Students were offered the Welsh translated paper alongside the English version.

Results

There are 99 fluent Welsh speakers throughout the 5 years and 68 who report that they are Welsh speaking but not fluent. There are 12 fluent Welsh speakers in Year 1 and 25 reporting that they have some Welsh skills. In 2016, only 5 of 300 students requested a translated paper, though we expect this to grow as the opportunity becomes better known.

Conclusion

We plan to repeat this in 2017, improving delivery following feedback. We have no plan to extend beyond Year 1, since students then should need less support in developing bilingualism; also, there are practical and financial challenges in conducting a parallel clinical examination in the medium of Welsh.

Student-generated electronic flashcards for the learning and assessment of basic science concepts

  • Dr Emma Taylor, University of Exeter Medical School
  • *Taylor E, Zamani R & Gallon C

Students undertaking Problem-Based Learning (PBL) courses are often uncertain about the level of detail required in their self-directed study. In many cases, students write large swathes of text gleaned from textbooks or journals, with little synthesis or clear understanding of the material. We introduced concept maps and electronic flashcards to our PBL sessions, to frame learning and encourage deeper understanding of basic science concepts. Here we examine the effectiveness of this approach.

For each PBL ‘trigger’ in a Year 2 Medical Sciences module, students generate electronic ‘flashcards’ to annotate each node of a pre-supplied concept map. Three flashcards - one formative, two summative - are submitted for staff and peer assessment, then released with feedback to all students as revision resources. This scheme was piloted in 2013/14, with formal assessment introduced in 2014/15, replacing a scientific essay. We compared short-answer question (SAQ) examination data from 2014/15 (n=48; post-flashcards) to 2012/13 (n=24; pre-flashcards).

Mean scores (56.3±15.2% vs. 44.3±14.6%; p60% (43.8% vs. 17.2%) increased in a previously low-scoring SAQ assessment after the introduction of flashcard assessment. No difference between these two cohorts was observed in the first year equivalent module that did not use flashcards (p=0.205, unpaired t-test).

A successful flashcard requires engagement with numerous key scientific skills: identification of key points, concise summarisation, and clear and visually appealing presentation of information. Students foster a deeper understanding of the topic by explaining it in their own words, while peer assessment provides ‘assessment for learning’ opportunities and plentiful revision resources.

Early evaluation suggests that the flashcards have been successful in promoting deeper, more active learning, and improved examination performance. Further evaluation will be carried out with upcoming cohorts.

Validation by design: applying a validity framework for item development

  • Mr Neville Chiavaroli, University of Melbourne
  • Corresponding Author *Chiavaroli, N.

The use of MCQ assessment in higher education is widespread, and there is no shortage of easily accessible guidelines for developing effective questions. However, effective development requires that question writers understand and appropriately apply the broader underlying principles of effective assessment to their own discipline, rather than attempting to merely comply with published guidelines. Furthermore, while the importance of collecting and interpreting validity evidence post-assessment is widely recognised (eg Cook et al 2015), validity considerations must not be allowed to become an afterthought, when it may be too late to correct crucial errors of design or undo the negative consequences of poor assessment.

The response of the presenter’s medical school has been to explicitly structure a validity approach to assessment around a three-fold process, namely: ‘internal’(acceptance and understanding of current MCQ writing guidelines), 'external’(peer review), and psychometric (interpretation of item statistics prior to reporting). This approach has been developed and evolved locally over several years of practice and review, and can be seen as an attempt to ‘build in’ validity to test development (van der Vleuten et al, 2012).

There remain several pedagogical challenges of this approach, including the challenge of adapting MCQs to assess higher cognitive levels of learning, and faculty developers and academics critiquing items in a colleague’s domain of expertise. More recently, we have focussed on redressing a comparative neglect of the validity implications of reporting results and delivering feedback to students. Overall, this presentation will argue that a critical aspect of validation involves consistently creating the optimal conditions and processes to support the inferences we make regarding our students’ learning.

Assessing a medical curriculum across three continents: reflections from establishment of a new medical school

  • Sean Hilton, University of Nicosia Medical School
  • Corresponding author *Charalambous A, *Hilton S

We report on implementation of a single UK MBBS course, franchised to a University in Cyprus, with clinical placements in Cyprus, Israel and the USA. Standardised assessments of knowledge, clinical skills and of longitudinal assessments for professional behaviours are delivered at three separate sites.

In the St George’s, London (SGUL) MBBS4 course, assessment matches learning outcomes domains specified in the UK GMC’s guidance Tomorrow’s Doctors 2009: Doctor as Scholar and Scientist; as Practitioner; and as Professional (DaP).

End of year exams knowledge and clinical skills. Throughout years 1 and 2, DaP is assessed using multiple indicators of professional behaviours e.g. attendance, contribution to small group work, reflective writing. For clinical attachments and in years 3 and 4, these are enhanced by overall professionalism assessments (based on the AAPediatrics rating scale) and a workplace based assessment (WPBA) portfolio. Students are placed at one of three sites - Cyprus, Tel Aviv or Chicago - for 3rd and 4th years. Faculty training for assessment takes place at each site.

UNic has now admitted five cohorts, totalling approx. 550 students, and the first cohort graduated in May 2015 (All 28 passing). Students come from >50 countries (largest proportion from North America). The challenges of delivering written exams and OSCEs, and assessing professionalism in these groups across three different healthcare structures and cultures will be discussed. Numerous challenges have been encountered to date, including lengthening the QC process to involve moderation and contributions from all clinical sites. Fitness to practise procedures have been invoked in a small number of cases.

Delivery of the course to a standard meeting the UK GMC’s QABME process has been achieved. Quality management processes are onerous but essential. Longitudinal assessment of professionalism is feasible in a wide range of learning environments and cultures.

Supporting international test-takers’ preparation for the BioMedical Admissions Test: A case study of BMAT in The Netherland

  • Dr Sarah McElwee, Admissions Testing Service, Cambridge Assessment
  • Corresponding author *McElwee S, Shannon M, Cheung K

Admissions tests, such as the BioMedical Admissions Test (BMAT) are used by medical schools alongside other selection criteria including interviews and work experience to differentiate between applicants. BMAT measures aptitude for demanding, science-based study by assessing the application of scientific knowledge, thinking skills, and written communication skills and is used by a range of UK and international universities to select for biomedical courses.

A revision of BMAT Section 2 (Knowledge and Application of Scientific Principles) was recently undertaken, with the specific aim of improving the extent to which BMAT could support widening participation and international candidates. The revision had two main WP aims: to clarify and more precisely define the test specification and content of BMAT to support candidate preparation; and to develop comprehensive and accessible free student preparation materials. An additional aim of the work in making the test specification more explicit was to support international students’ preparation and ensure that where BMAT was used internationally, the content of this section was targeted appropriately. The revision process will be outlined and the challenges of aligning the development with best practice principles for fostering positive washback ‘additional benefits for students in the knowledge and skills they develop through preparation’ will be discussed.

Senior academics from stakeholder universities collaborated on this project. In particular we will focus on the international dimension of BMAT and the work done by Dutch subject matter experts and the Admissions Testing Service (ATS) to prepare for BMAT’s use in The Netherlands. ATS worked with a carefully chosen publisher to produce a free ebook revision guide for BMAT Section 2; evidence from online forums suggest it was received very positively by candidates. Both the specification definition and revision guide helped clarify the scope of topics for candidates to revise, and made core preparation materials freely available to support equality of access.