Maths CAA Series: June 2002
Page Guide: Home > Articles > maths-caa-series >
|
Abstract |
|
Introduction |
|
The first year; a community of practice |
|
The follow on |
|
The present position |
|
Reflections on the lessons learnt |
|
And finally for the future: accessibility |
|
References |
Abstract
This article describes the use of CAA in a statistics service course, and its gradual evolution over 4 years, from a basic but surprisingly encouraging start, to a more sophisticated system which now faces the imminent challenge of accessibility issues.
Introduction
Economic Statistics and Computing is a compulsory double module for all first year students of economics, and as such it is fairly typical of a service teaching module, though smaller than many with approximately 70 students. The statistics teaching is delivered by weekly lectures, eight computer workshops, and is supported by the Mathematics Support Centre.
A problem of students failing to engage with the work was initially addressed in 1992 with the introduction of a student logbook, containing checklists of concepts and questions, relating to each week’s material. Answers were provided at the end of each section so students could self assess their progress and competence. Students were encouraged to work through the relevant section each week, seeking help where necessary in the Mathematics Support Centre, and at the end of term sat a test consisting of questions very similar to those in the logbook. This was not a great success, test marks remained low, and it was clear that students were failing to grasp essential concepts.
The implementation of WebCT as an online learning environment at Coventry was the ideal opportunity to provide 20 weekly online assessments. Anything which would provide students with an opportunity to learn, whilst being assessed for marks, and not need any marking by hand, was worth a try. Thus in 1998, a year before the University implemented WebCT across the campus, we made the foray into CAA, and in the ensuing 4 years have refined the model and the approach, in the light of experience which has been both painful and pleasurable. Computer Aided Assessment is, though, only part of the diet of assessment for these students. Initially accounting for 25% of the statistics mark, it now accounts for 40%.
The first year: a community of practice
The initial strategy was to provide weekly tests on that week’s lecture, available for only 2 weeks from the date of the lecture, with only one attempt allowed and no feedback. The aim was to engage students in their work, so questions were deliberately kept short and relatively simple. WebCT allows for a wide variety of question types; multiple choice, matching, short answer, and in particular calculated questions, and the ability to incorporate images and a feedback comment. The tests were not identical, but there was only a very limited number of questions available. They were open book tests and there was no supervision. In practice, and a good one, students did them at the end of their IT workshops, and, doubtless helped each other.
Unfortunately I could not keep up with the pace or the problems. This brought home to me the first lesson in successful CAA: it takes a lot of time to write good questions in sufficient numbers. The second home truth was that being at the cutting edge sometimes cuts, but the third truth was that, handled positively, an apparent failure can be turned into something approaching a success.
Surprisingly the problems we encountered motivated the students, which was the whole point of the exercise, though not an outcome I had anticipated when faced by such difficulties. The students appreciated being the first to try out new technology, and when tests appeared were competitive to be the first to find the week’s bug, or my mistake. The fact that their emails reporting bugs were replied to promptly built up a good rapport, and they gave much encouraging feedback.
In the previous year only 12% had achieved 70% or more in the end of term tests. With WebCT, using questions which were largely based on those that had already existed in the logbooks, 42% gained 70% or more over all the tests. 65% of those surveyed felt that the tests had taught them something, and approximately a third indicated that they actually enjoyed doing the tests.
Here was summative assessment that turned out to be formative, and to have encouraged learning in a fairly pleasurable manner. It provided students with a relatively painless process of acquiring knowledge and marks, was an introduction to web based learning, and provided the incentive for me to attempt to develop good web based materials.
On reflection, the success of this initial experience may well have had more to do with the ‘community of practice’ that we in effect established; an illustration of Wegner’s social theory of learning. Wegner proposed that students need a place of engagement, experiences with which to build an identity and ways of making their actions matter. In coming together to test out the early implementation of on line testing in WebCT we provided those three factors. This is a design principle that I have borne in mind since, but have failed to recapture in a similar manner, partly as a result of CAA, and WebCT, having become more established throughout the University.
The follow on
In 1999/2000 a similar pattern of testing was adopted, but with a greater number of questions, which ensured that individual tests were very different from each other. Feedback was provided for some, but not all questions. In that year student comments indicated that 50% of the students did the tests regularly, and 60% of them felt they had learnt through the tests. Almost 50% of them enjoyed, yes enjoyed, doing some or all of them and no-one had a negative comment about the use of the computer testing system.
By now CAA was well established as a form of assessment within this module, and as less attention needed to be paid to the mechanics of its delivery, more attention was required in the area of its purpose and form. Biggs’ view is that "What and how students learn depends to a major extent on how they think they will be assessed." CAA , within a wider assessment framework, can offer students the opportunity for formative, as well as summative assessment, with useful and timely feedback, and the opportunity to practice simple calculations, and view the subject within a wider context. Thinking of the tests within this framework helped identify the key concepts of each lecture and the areas in which students might need practice.
Timing was also of importance. Web based tests are not limited to time or place. Students could theoretically access them at any time and in any place with an internet connection, at their convenience, though in a University less blessed with good computer provision this might be a problem. By limiting the availability of tests to just 2 weeks after each lecture, students were being encouraged to keep up with the course, and to prepare themselves with the basic knowledge before tackling the extended assignments. Students who were not engaging could be identified, by checking whether or not they had done the test, and some early remedial action initiated. Test responses also provided information to the lecturer on how well concepts were being grasped, and on whether the questions were discriminatory enough.
Students were encouraged at the end of each test to email the lecturer if they had any queries or difficulties. The aim was to remove the complete impersonality of the test. This resulted in relatively few emails, but over the course of the two terms many different students responded once or twice.
In the third year of operation students were offered two attempts at each test, with feedback, and with the higher mark counting for assessment. There were sufficient questions available to ensure that the second test was not identical to the first, though the pattern of questions was similar. This had the advantage of giving students a practice run and some familiarity with the format of each test before having to do it ‘for real’. Use was made of self tests, which were multiple choice exercises that could be repeated ad infinitum, with detailed feedback, and the on line course materials included some interactive questions with drop down boxes providing the answers to stages of a calculation.
The on line assessment at this stage amounted to 25% of the statistics mark, and the primary aim was still engagement. As the tests were unsupervised the opportunity for collaborative work and impersonation was unlimited. Collaboration, known as group work, is of course actively encouraged as a graduate skill. Long after the event, a cheerful student hinted that, as students became desperate for good marks in tests to pass the module, he had done rather a large number of tests for his friends and was getting quite good at them.
The present position
In view of the remarks of the cheerful student, and of a review of the economics degree in 2001, it was decided to introduce some supervised assessment for this module. So history is now repeating itself in on line form, with two supervised, open book, end of term on line tests, each worth 10% of the total marks, based on questions taken from the earlier weekly online tests for that term. 25% of students gained 70% or more over both tests, but more worryingly 20% gained less than 40%. Worrying in that assessment assesses the lecturer as well.
The weekly tests have changed in format, too, and are now worth 20% of the total marks. There is a practice test, which a student may attempt as many times as they like, and which provides feedback to help them take the correct approach next time. Practice tests are not entirely identical each time a student sits them, so there is some point to repeating them. In writing each test the aim has been to provide some easy questions at the outset to build confidence, but then to provide non trivial questions which need at least a pause for thought, if not paper, pen and calculator too.
The real test, however, can only be attempted once, and is time limited to 20 minutes. It is unsupervised and open book, and consists of very similar questions to the practice test, but there is no feedback and only the student’s response is shown, not the correct answer. Because of a large expansion in the question bank, and the facility in WebCT to select questions at random, no two students will receive anything like the same test. Great care was taken, of course, to ensure that in a set of alternatives for one question, the correct answer was not always in the same position.
By way of variation, and to exercise other desirable graduate skills, one test, on the RPI, was in a different format with two attempts offered for the same test that required a little research to answer. All students received an identical test, though all did not realise this.
History is repeating itself here too. In 1992 Logbooks should have been, but were not, completed. In 2002 only 10% of students did all the on line tests, and whilst almost 30% of students averaged 60% or more, 46% of students gained less than 40%, and no students now report actively enjoying the experience. One reason for this may have been a change of software in mid session, requiring students to complete a double log in process to access the new tests. This coincided with a period of difficulty in accessing WebCT from outside the University, which underlines the need for robustness in both hardware and software when embarking on CAA.
Reflections on the lessons learnt
Having lived with the experience for 4 years in a changing world of student abilities and experiences, I am convinced that within an overall teaching strategy the provision of regular on line tests is a very valuable resource, for both student and lecturer.
- Be challenged about the overall diet of assessment.
- Think clearly about the key concepts to be taught, and how they may best be assessed.
- Identify what is best assessed and practiced using CAA.
- Provide assessment which is valid, reliable, fair and useful to the student.
- Save time on marking, though depending on the size of the module this is more than offset in the first few years by the time spent on question writing.
- Track the progress of students, and identify early those who may be struggling.
- Monitor the understanding of concepts.
- Be creative, being the stimulus for the development of other web based materials.
- A ‘low level’ activity to encourage participation and engagement, whilst gaining marks
- Timely feedback at the time of need and interest.
- The opportunity for practising the application of concepts and skills, in practice tests.
- Self assessment
- Opportunities to show the lecturer that you are cleverer than they are, when you spot their silly mistakes.
The practicalities I have learnt:
- As everyone who has ever done it will say, it takes enormous amounts of time to write good questions in sufficient numbers to provide a variety of tests, but year by year these can be increased and improved.
- Read ‘Blueprint for Computer Aided Assessment’ by Joanna Bull and Colleen McKenna, and published by the CAA Centre.
- It does not have to be perfect the first time round; indeed many students respect lecturers more for acknowledging their mistakes.
- It takes time and patience to deal with software problems, and good technical support, or at least a similar mind, is a must.
- Hardware and software needs to be robust.
- Give students an opportunity to make a mess of at least one test while they find their way around the technology.
- In writing questions, which involve units or answers to a certain number of decimal places, give an example of the format of the answer required, so students are quite clear about what to type in.
- Ideally introduce tests in a workshop session in the presence of a lecturer to ensure that students safely navigate the initial hurdles of the technology.
- Plagiarism and impersonation exists; supervised tests and secure browsers are an answer.
And finally for the future: accessibility
The challenge now is to review the accessibility of tests in the light of the provisions of The Special Educational Needs and Disability Act, which will come into force in September 2002. These provide legal protection for disabled students in all aspects of educational life, and places on HE
- a duty not to treat disabled students less favourably, without justification, for a reason which relates to their disability; and
- a duty to make reasonable adjustments to ensure that people who are disabled are not put at a substantial disadvantage compared to people who are not disabled in accessing further, higher and LEA secured education.
This is not an easy challenge, and needs to be taken seriously by all of us involved in CAA. For further information see the TechDis website http://www.techdis.ac.uk/news.html . TechDis is a Joint Information Systems Committee (JISC) funded service supporting the further and higher education community in all aspects of technology and disabilities and/or learning difficulties.
[2] Bull, J and McKenna, C (2001), Blueprint for Computer-assisted Assessment, The CAA centre
[3] Wegner, E (1998), Communities of Practice: Learning, Meaning and Identity, Cambridge University Press
Maintained by R.L.Surowiec@bham.ac.uk
Last revised: 24 June 2002