The Kathleen Tattersall New Assessment Researcher Award

Interview with Dr. Paula Lehane, winner of the Kathleen Tattersall New Assessment Researcher Award for 2022

Interviewer: Congratulations Dr. Lehane, for being the winner of the Kathleen Tattersall New Assessment Researcher Award for 2022. Could you tell us a bit about yourself and how you became interested in digital assessments.

Dr. Paula Lehane: My name is Paula Lehane and I am an Assistant Professor in the School of Inclusive and Special Education in Dublin City University (DCU), Ireland. Prior to this, I was a primary school teacher in a developing school in Dublin and it was during this time I became very interested in inclusive education, which led me to complete a Graduate Diploma in Special Educational Needs and a Masters in Additional Support Needs. However, I was also very drawn to the field of digital assessment too. While working as a teacher, I noticed the poor quality of many digital diagnostic and screening assessments commonly used to identify specific educational needs e.g. dyslexia. I was most concerned as to whether these assessments were actually measuring what students really did know and whether their design was interfering with the assessment process. I decided to take a career break to investigate further and joined the Centre for Assessment Research, Policy & Practice in Education (CARPE) at DCU, to work on issues related to assessment and test development in educational and workplace settings. While there, I completed my PhD in the field of digital tests and assessments for post-primary learners.

Interviewer: Your research interests today encompasses assessment, inclusion and technology. How do these concretely inform policy and practice in educational assessment?

Dr. Paula Lehane: With the broader and in-depth understanding I have gained from my research on digital assessments, my goal is to be able to effectively link research, policy and practice to affect positive change in our education’s assessment processes. For example, some of my work at CARPE informed the development of the very first computer-based exams (in the area of Computer Science) used in Ireland’s Leaving Certificate in 2021, our terminal assessment for post-primary learners. These exams are very high-stakes as they can determine how students are admitted to college in Ireland. I carried out a literature review that informed policymakers, test developers and teachers about the potential design and deployment issues that should be considered for the newly-introduced computer science exam. Although the exam is still in its infancy stages, there are no major kickbacks or incidences as a result of this research-informed approach to introducing digital assessments to an established system.

In my PhD research, I explored the use of pictures, texts and animations in PISA items to understand how test-takers were engaging with digital assessments. For example, I examined what differences occurred in students’ engagement and performance with test items if they were invited to listen to an audio narrated animation rather than reading a piece of text. Would this particular design provide a better understanding of their current abilities, knowledge and skills? Combining quantitative with qualitative data from post-primary students, I gained much insight into how these test-takers were behaving in an assessment with these design features. Students watched a replay of their eye movements and reported what they were thinking at different parts of the animation. This allowed me to capture the nuances of how students work when multimedia stimuli are involved in testing contexts. In terms of test design, students also reported that they appreciated the PISA-style assessment interface but commented on their limited freedom to navigate back and forth between questions.

Another key finding from that research, which aligns with the experiences of other countries too, was the importance of students to be familiar with the assessment interface and types of assessment activities prior to a high-stakes exam. Students would then be capable of navigating within an online test environment that they know well, and choosing strategies that work instead of wasting time (and getting stressed!) in a high-stakes exam trying to get used to the interface.

Interviewer: Going forward, what do you think are some key considerations for policymakers when developing and implementing digital assessments?

Dr. Paula Lehane: I think that we need to recognise that if we are going to transition fully to the use of computer-based exams in post-primary settings, we should avoid adopting a technocentric mindset. Therefore, it is essential to pilot a digital assessment program, double-check it, and engage early with the test-takers themselves and not assume that we know how digital assessments work for different learners. Such a dialogue between researchers, test-takers and policymakers can facilitate the successful deployment of digital assessments in a systematic, effective and fair manner that does not disadvantage certain groups or advantage others.

Interviewer: Going forward, how do you think the Kathleen Tattersall Award is going to help you? And what message would you give to new researchers?

Dr. Paula Lehane: I realised when starting my research that education systems across the globe are all grappling with the same issues regarding the role of digital technology in pedagogy and assessment. This award is a great way for me to share my results with, and talk to, other researchers, practitioners and policymakers who are tackling the same questions and ideas. Talking to the AEA-Europe community about these topics is an exciting opportunity for me to learn from international peers.

Looking to the future, I plan to explore the ideas of equity, accessibility and digital assessment and later also transition into the field of teacher assessment literacy.

I would very much encourage researchers who are thinking of applying for this award to do so. After all, research is not meant to be locked in an office; and the Kathleen Tattersall Award is a really effective way of getting your work into rooms with like-minded people who will not only advance your own current thinking, but who will also help you reflect on where you want to go next.