School of Nursing and Midwifery, Queen's University Belfast, Belfast, UKUnited Kingdom
Learning and Teaching Support Division, Information Services, Queen's University Belfast, Belfast, UKUnited Kingdom
Also available as PDF (figures and tables appear in the PDF).
There is a dearth of evidence focusing on student preferences for computer-based testing versus testing via student response systems for summative assessment in undergraduate education. This quantitative study compared the preference and acceptability of computer-based testing and a student response system for completing multiple choice questions in undergraduate nursing education. After using both computer-based testing and a student response system to complete multiple choice questions, 192 first year undergraduate nursing students rated their preferences and attitudes towards using computer-based testing and a student response system. Results indicated that seventy four percent felt the student response system was easy to use. Fifty six percent felt the student response system took more time than the computer-based testing to become familiar with. Sixty Percent felt computer-based testing was more users friendly. Seventy Percent of students would prefer to take a multiple choice question summative exam via computer-based testing, although Fifty percent would be happy to take using student response system. Results are useful for undergraduate educators in relation to student’s preference for using computer-based testing or student response system to undertake a summative multiple choice question exam.
Keywords: Computer-based testing; Student response system; Summative assessment; Undergraduate nursing education; Computer-based testing; Student response system; Summative assessment; Un- dergraduate nursing education
Student response systems, sometime known as ‘class response systems’ or ‘clickers’ have been used in education for over 35 years.1 Indeed in terms of health care education authors have reported their usefulness in nursing2 dentistry3 radiology4 and medicine.5 In a University setting, they have been used with a variety of learners, from undergraduate to postgraduate students. Nonetheless, it has been noted that the majority of the existing literature in relation to their utility is anecdotal and there remains a clear need for rigorous exploration on the use of this technology.6 This is particularly true for the role of student response systems in summative assessments.
From the literature the advantages of using student response systems in teaching have been delineated.7 It has received a positive review from students and enhanced their learner engagement and participation.8 Within nursing education, student response systems have been shown to: increase classroom engagement 1; provide more effective and efficient nurse educa tion8; and students themselves have highlighting that they help to improve acquisition and Page 11 retention of knowledge. 9 However, while positives have been described, there are limitations to using a student response system which have also been noted. These include the fact that the corresponding receiver must be connected to and recognised by the computer prior to the software being started, otherwise responses will not be collected. Additionally, there must be technical familiarisation with the system to ensure there is minimal distraction from the course of study.10
This is the first year that a novel module on Evidence Based Nursing (EBN1) has run in the current undergraduate nursing education curriculum within the recruiting University. What brought the authors to this topic was the need to select a useful format to conduct summative assessment. In this instance this refers to the summative year 1 EBN1 computer-based testing of a multiple choice question exam for undergraduate nurses (approximately 450 per year). The students are divided over two intakes per year, approximately 350 in October (including adult, children, mental health and learning disability specialism) and approximately 100 in February (adult specialism only). In some cases, due to the lack of a computer suite to accommodate the larger student numbers, all students cannot take a multiple choice question exam at one time and half must wait in a holding room until a computer is available, meaning the time is doubled for the exam. Module coordinators (JR and CL) sought an alternative, but prior to uptake recognised the importance of testing such a system as student response system. To this end they conducted two formative assessments (one using computer-based testing and one using a student response systems) and evaluated preferences for both from the students. The role of using student response systems to improve active teaching and learning and for formative assessments is well documented. 1,8-12 However, using student response systems for summative assessment is sparsely reported within the literature13 and there is a dearth of evidence on student preferences for computer-based testing versus student response systems for summative assessment.
Student response systems are small hand-held devises that can be used either anonymously or a student can authenticate them selves by entering usually their student/exam number, to collect individual responses for exam purposes. They allow students to answer questions in real time. 14 A Universal Serial Bus (USB) radio based receiver attached to a classroom computer collects all data including the responses provided. The benefit of engaging students with student response systems in teaching has previously been discussed. 1,8-12 However this paper will focus on student’s preferences for using student response systems, compared to computer-based testing, in summative examinations.
Computer-based Testing is conducted within the recruitment site through a University student homepage. The assessment is accessed through the module homepage from within the Univer sity virtual learning environment. This multiple choice question assessment is scheduled for each individual student and is only available for completion during a predefined time period. Students are required to attend a University computer suite, log into their student account and complete the multiple choice questions assessment in an exam invigilated environment.
This study is based on quantitative data collection and analysis.
During the academic year of 2014-2015, the EBN1 module was conducted in year 1 of the undergraduate nursing degree programme at a School of Nursing and Midwifery, taught in a large University within the United Kingdom. Data for this study was gathered from the October 2014 intake of students (n=324).This intake comprises adult, children, learning disability and mental health nursing students. The module comprised lectures, an online e-resource and small group tutorials. The module coor dinators (JR and CL) standardised all teaching material for the small group tutorials to ensure all groups covered the same material in an identical fashion. At the end of phase 1 (November 2014) we asked students to complete a set of formative multiple choice questions via computer-based testing. At the end of Phase 2 (February 2015) we asked students to completed a set of formative multiple choice questions via a student response systems. Following the formative assessment completed by the student response system at the end of Phase 2, we asked students their opinion and preferences of computer-based testing and the student response systems (Table 1 presents the statements asked). All data was analysed using descriptive statistics. Prior to conducting this study, ethical permission to conduct this work was gained from The School Research Ethics Committee of the School of Nursing and Midwifery, within the recruiting University.
Students accessed the multiple choice questions via Question Mark software on their chosen computer (home or university) at a time convenient to them. Immediate feedback was given to all students on completion of the formative assessment.
Students were located in a lecture theatre and given a student response systems (Turning Point NXT) handset and paper copy of the multiple choice questions which they worked though individually using self-paced polling.
In planning the evaluation the aim was to explore the more convenient and user friendly option for students when undertaking Page 12 a multiple choice question exam. We asked 6 questions based on this to all students in the October 2014 intake. The questions were on: ease of use; format of delivery of question; preference for use in a multiple choice question exam; user friendliness; and time to become familiar with technology.
Of the October 2014 intake of undergraduate nursing students (n=328), 210 students signed into actively participate in the evaluation. Participants who answered the questions asked ranged from 192- 157. Table 1 below details the questions asked, answer choices provided and the number of participates who responded.
In assessing the ease of use of the student response system handset (Question 1, respondents=187) 74% of respondents either strongly agreed (37%) or agreed (37%) that the handset was easy to use (12% undecided) with only a minority disagreeing (8%) and strongly disagreeing (5%). The format of the question when using the student response system is that the questions are provided on written sheet and the students’ works thought the questions at their own pace (self-paced polling). On asking if this format enabled students to engage with the multiple choice questions (question 2, respondents=182) the majority of respondents felt it did, with 27% strongly agreeing and 32% agreeing. 21% of respondents were undecided with 20% disagreed that the format enabled them to engage with the questions (9% disagreeing and 11% strongly disagreeing). On asking the students which mode takes more time to become familiar with (question 3, respondents=157) the majority of students who answered (56%) felt the student response system took more time or both the student response system and computer-based testing system took the same time to become familiar with (33%), with only 11% answering that the computer-based testing system took more time for them to become familiar with. On asking if the student response system or computer-based testing system was more user friendly (question 4, respondents=190) 60% of those who responded felt the computer-based testing system was more user friendly. 27% preferred the student response system with 13% having no preference for either system. In relation to this study, we were particularly interested in the possibility of using the student response system for future summative multiple choice question examinations. We therefore asked students if they would prefer to take a multiple choice question exam using computer-based testing or student response system (question 5, respondents=174). 70% of those who responded stipulated they would prefer to take via computer-based testing, with 30% preferring to take via student response system. Nonetheless, 50% of students would have been happy to take a multiple choice question exam using the student response system (question 6, respondents=192) with 28% strongly agreeing, 22% agreeing. 30% of those who responded disagreed (14% disagree, 16% strongly disagreeing) and would not have been happy to use the student response system to complete a multiple choice questions, with 20% of respondents undecided.
In this article, the authors examine the preferences of year 1 undergraduate student nurses in using computer-based testing and a student response system for multiple choice question ex- Question Answers Number of Responses 1. The SRS was easy to use Strongly agree – 37% Agree – 38% Undecided – 12% Disagree – 8% Strongly disagree – 5% 2. The format of the questions in the SRS assessment enabled me to engage with the questions Strongly agree – 26% Agree – 32% Undecided – 22% Disagree – 9% Strongly disagree – 11% 3. Which takes more time to become familiar with?
Student response system – 56% Computer based testing – 11% Both take the same time to become familiar with – 33% 4. Which is more user friendly?
Student response system – 27% Computer based testing – 60% No preference – 13% 5. If you were to take an MCQ summative exam, which mode would you prefer to use?
Student response system – 30% Computer based testing – 70% 174 6. I would be happy to use the SRS in an exam setting Strongly agree – 28% Agree – 22% Undecided – 20% Disagree – 14% Strongly disagree – 16% Table 1: CBT/SRS Evaluation questions and responses.
Page 13 aminations. This study has limitations and it is important that the findings are viewed in light of these. The study is limited to only one group of students. Furthermore, while there were over 300 students within this group, those who responded to questions for this evaluation ranged from 157-192. Thus, findings may not be representative of the whole cohort. Nonetheless, this study is novel as it has exposed the same groups of students to multiple choice question formative examinations via both computer-based testing and a student response system and explored preferences for mode of examination. The role of using student response system in formative assessments is well documented1,8-12 but there is a paucity of literature currently available examining student preferences for computer-based testing versus student response systems for summative assessment.
The student response system is a relatively new system within the University and was never used within EBN1 prior to asking students to undertake the formative multiple choice questions. Thus, student’s familiarisation with this system, as opposed to computer-based testing which is used regularly throughout the module, is expected to be much lower. It would therefore be worthwhile integrating the student response system into lectures and tutorials to engage the students with the system and increase familiarisation with it and repeat the evaluation. Indeed the value of integrating such methods of active learn ing into education has been previously documented. 1,11,15 The authors intend to do this with the future intakes of students. The availability of a single computer suite, which is sufficiently large to house over 300 students who need computer access for a multiple choice question examination, is a major factor in choosing a student response system or computer-based testing for a summative assessment. In an already busy University examination fortnight at the end of each year, the availability of such a room is not always met. Indeed, for previous multiple choice question exams half of the intake of students have sat an multiple choice question exam with the other half in a ‘holding room’ so that they cannot discuss the test with those who have taken it, until the first half have completed. For the second half of students this means the exam period takes twice as long and they must wait in exam conditions for half to complete the exam prior to them sitting the exam. Considering a multiple choice question exam can be conducted with a student response system in any classroom which can invigilate 300 students, is a major advantage to both students and invigilating staff’s time and could make optimal use of University resources. In relation to using computer-based testing or a student response system as opposed to paper testing for multiple choice questions there are two main advantages. Firstly, the tests are marked automatically with both computer-based testing and student response systems as opposed to being marked by hand by academic staff or hand fed into a machine readable answer sheet. Secondly within a University setting, old tests cannot simply be thrown away, if the multiple choice question exams was paper based this would mean the storage and then disposal of copious answer sheets. By delivering a multiple choice question test either by computerbased testing or a student response system, it eliminates this expense.
This study has used quantitative methods to compare the preference and acceptability of computer-based testing and a student response system for completing multiple choice questions in undergraduate nursing education. Study findings indicated that the student response system was easy to use but took more time than the computer-based testing to become familiar with. The majority of students indicated they would prefer to take a multiple choice question summative exam via computer-based testing, although half of those surveyed would be happy to take using a student response system.
Technology is a fast and evolving medium in higher education and educators must investigate the pedagogical value of new development to establish their usefulness for undergraduate education. The integration of regular student response systems usage into lectures alongside current computer-based learning activities is needed to ensure students are familiar with both forms of equipment. After this, further research is needed to ascertain which mode is more acceptable to students for a multiple choice question examination. The accessibility, convenience and implication for University resources, particularly examination timetabling, that could come from using the student response system for examination purposes underscores the importance of such work. Nevertheless research must demonstrate students’ receptiveness and acceptability of using a student response system prior to it being implemented.
CONFLICTS OF INTEREST: None to declare.
The authors obtain written informed consent from the partici pants for submission of this manuscript for publication ETHICAL APPROVAL STATEMENT Prior to conducting this study, ethical permission to conduct this work was gained from The School Research Ethics Committee of the School of Nursing and Midwifery, within the recruiting University.