Maths CAA Series: September 2002
Page Guide: Home > Articles > maths-caa-series > |
Introduction |
|
The value of computer-based assessment |
|
Problems and Limitations |
|
Discussion and Conclusions |
|
Acknowledgement |
|
References |
Introduction
The Mathwise project [1-3] began in the early nineties as a major collaborative undertaking, involving many UK Mathematics departments. In the early days the main focus was on the development of learning materials, in topic-based modules. Assessment sections were added "in earnest" to Mathwise modules mainly by the SUMSMAN project [4]. Although the uptake of Mathwise was quite widespread through the mid to late nineties, this subsequently tended to fall away, particularly with the advent of the Internet. Mathwise is a CD product, so that accessibility can be a problem for students, compared to Internet tools. However, the general approach adopted by Mathwise is robust and is capable of advancing with developing technology.
This article sets out to reflect on experiences with Mathwise during the ten years or so of the project, discussing lessons learned, the limitations of software and the likely shape of future developments. It is vitally important that the Mathematics community should not forget the Mathwise experience, but rather carry forward the lessons learnt, to inform the future development of mathematical Computer-Based Learning and Assessment, which will inevitably be Web-based.
The value of computer-based assessment
What benefits then has the Mathwise assessment software brought? There are some fairly obvious advantages in all computer-based assessment, such as a saving in staff marking effort, instant feedback to the student, and even an environmental bonus of saving paper!
Some further advantages of all computer-based assessment are:-
- The absence of a human marker means that computer-based assessment is perhaps the ultimate form of anonymous marking. This does help put the student at ease whilst doing a test, because it removes the risk of looking foolish to a human marker.
- The computer provides a structured environment in which the student can practise and do questions. Sometimes a student can struggle with tutorial exercises in a textbook, because he does not quite know where to start. The computer on the other hand structures questions, so that at least the student can make a start. In discussion one of my students said Mathwise assessment was a kind of "crutch", but a useful one. Perhaps it is analogous to the practice of putting stabilisers on to a child’s bicycle to help build confidence in the early stages.
- Most Mathematics textbooks provide answers at the back. Students seem unable to resist the temptation of looking at them before tackling the question! On the computer, this temptation is removed, so that the student is forced to make a genuine attempt before receiving any feedback.
Mathwise also has a number of particular advantages:-
- Mathwise uses random parameters in its questions, so that the student can neither copy nor simply memorise what the answer to a question was. However, the student can return to the equivalent question to try it again in a meaningful way. A further bonus is that a resit test is very simple to organise, in the case, for example, when a student is off sick on the day of the test. Evidence gathered suggests that students spend a lot of time practising on Mathwise and usually obtain good marks on the tests.
- Mathwise can mark questions in algebraic form, so that students can tackle calculations of substance.
- Mathwise has the facility to browse questions before deciding in which order to tackle them.
- Mathwise has facilities for mathematical input and for partial credit, although there remain significant issues with these aspects.
Problems and Limitations
Limitations we have encountered with Mathwise and related software have been discussed previously [5]. Limitations of Mathwise assessment as it stands include the following, although all these could have been overcome if more funding had been obtained for development. It would have been simply a case of extending and building on the overall approach adopted.
- Mathwise lacks a facility to enable the student to revisit a question and review answers, as in a conventional examination.
- The feedback given by Mathwise is very limited. The student is merely told which Mathwise learning unit to look at, with no further comment.
- The Mathwise question bank is limited, mainly because of the high cost of authoring questions in the format provided. As a result there has been a tendency for students to "spot" questions and anticipate correct answers by means other than calculating in the way intended. Of course this can and does happen in conventional examinations, when "banker" questions can be predicted, but even so a computer-based system should preferably eliminate this. The solution to this problem is simply to author more questions.
- A related problem is that the Mathwise tests are all limited, each to one topic. There is the Differential Equations test, the Complex Numbers test and so on. It would be good to have the flexibility of being able to construct one’s own test, pulling in questions from different topics.
There are however problems and limitations with all computer-based assessment. A number of the problems we have encountered with Mathwise that appear to fall into that category include:-
- There is an inherent problem associated with typing mathematical expressions as input to a computer. Although Mathwise and other systems do provide a variety of input tools, there appears to be no way of avoiding the fact that we write Mathematics using symbols that do not lend themselves readily to single line input. An unfortunate consequence is that on a number of occasions students have commented that the computer marked them wrong, when they claim to have obtained the correct answer. In these cases it appears that the student has made a mistake due to a typing error, even in spite of the input tool being available.
- Partial credit is a thorny problem, as discussed in [6]. Although Mathwise does include facilities allowing the student to break a question down into several stages, somehow students can still find this problematic. The difficulty perhaps is that the computer, rather like a benevolent dictator, imposes its own format on the student. It breaks the question down in the way assumed appropriate by the person setting the question, which may not have been the way the student would have tackled it.
- There can be problems with the wording of questions on the computer. When translating an examination question into a form that is programmable into a computer-based assessment system, some changes can be necessary. For example in one question a student was differentiating (5x3+3x)ln(4x+2) successfully. However, one key step in Mathwise confused him. The intention was to test whether the student was familiar with the product rule, and the key step said, "Derivative of uv is u’v + puv’ with p=?". This phrasing confused the student and the class tutor intervened and explained to the student what was meant. Although the student could do the question, the wording in this particular key step confused him.
- A problem arises simply because a computer cannot apply discretion and common sense as a human marker would. In one question the student had "essentially" answered a question correctly, but simply mis-typed one character, entering sin(4x+3) in a part of the answer when he had intended sin(4x+1). The computer here dispassionately awarded a mark of zero for the whole question, when a human marker might have used discretion and awarded some marks.
Discussion and Conclusions
How then can we summarise the lessons learnt about Computer-Aided Assessment through the Mathwise experience? I conclude this article with some observations.
Assessment is a very important part of learning. In Mathematics students learn a lot by doing tests and having them marked. It is fair to say that it has been mostly via its assessment sections that Mathwise has been used meaningfully by students. In class observations I have noticed how students will often find the assessment parts of Mathwise and spend a lot of time in them, even though they were not explicitly directed there. They like to be given a specific task to do, particularly one that gives a "score". This is the "Nintendo effect" mentioned in [7]. In a similar system, Statwise [8], I have also observed how students sometimes switch out of a test and quickly back into learning materials to check a detail. This confirms the formative effect of Computer-Based Assessment in general.
As regards the Mathwise learning units, although they contain well constructed multimedia sections, they are text-based and the tendency has been for many students (although by no means all!) to skim through the material superficially, without engaging deeply enough with the content being presented. I have analysed student questionnaires, designed to establish how "deep", "strategic" or "surface" students are in their attitude to study. Results suggest that students tend to become rather strategic in their approach at university. The question most frequently asked by many students is perhaps, "Does this count towards my mark?" Although we may well lament this tendency on idealistic grounds, a key advantage of Computer-Based Assessment is that it engages with the bulk of students at the point where they actually are in their development. It provides an activity which identifies with a strategic approach, but which also obliges students to work through mathematical detail very carefully. It seems to me that this is the essence of good teaching. Computer-Based Assessment is valuable in motivating students to do calculations and in providing a first experience of Computer-Based Mathematics materials in the early years of their studies. As students use Mathwise assessment, they quickly learn the importance of practice as a means of getting an improved mark. The randomisation of parameters in Mathwise assessment makes repeated practice a meaningful and useful exercise. It is difficult to argue with an activity which prompts students to practise mathematical test questions repeatedly.
Although it is true that there are some significant problems with input, partial credit and wording, these should not be over-stated. The input issue arises in all mathematical software packages requiring single line input. It could be argued that the acknowledged problems with partial credit are counter-balanced by the fact that the Mathwise key steps do show the student ways of breaking a question down into smaller steps. This is helpful because it can often happen that a student simply does not know how to begin tackling a mathematical question. Formatively the Mathwise key steps can act almost like a tutor, prompting the student with ways of breaking the question down into smaller steps. As for wording, it is always the case that things are worded somewhat differently in different media. The same message or question might well be phrased a little differently on television than in a newspaper article, for example. It should not surprise or alarm us unduly therefore that adjustments are needed to the wordings of questions translated on to the computer from paper-based tests.
The question has been asked, how Computer-Based Assessment compares with paper-based tests. We asked students and found that about a half of them felt there was no difference, a quarter felt they would have done better on a paper test, and a quarter felt they would have done worse [5]. This suggests that the two media are comparable overall.
The reality is that, when using any new technology to do familiar tasks, there is always a learning curve. It takes some time and practice to "get the hang" of how to use it appropriately. Many of the problems experienced with Computer-Based Assessment can be alleviated or solved by giving students induction sessions in which they learn how to use the system. In the early days of the motor car, problems were experienced with more people being killed in accidents, as compared to the days of the horse-drawn carriage. The solution to this problem was not to abolish the motor car, but to introduce a proper driving test and to provide lessons with skilled instructors.
If there is a "bottom line" with Mathwise, it is that students do in fact like it and enjoy using it. This has been a consistent theme from student responses to questionnaires regarding Mathwise and other similar systems [5]. I have observed that students often form pairs or threes spontaneously when practicing a Mathwise test. This helps engender a sense of community and peer help. Students are often good at explaining things to each other and encouraging each other. This in itself is a useful educational experience and confirms the observation [7] that Computer-Based Assessment helps provide a social context in which learning takes place. However, although students are usually quite relaxed in practice sessions, they tend to become rather more tense in "for real" sessions where their mark "counts", as we would expect
A further desirable feature of Mathwise is that it has provided a sense of community of authors and users among university Mathematics academics. It has been a focus of significant debate on educational practice. Mathwise was a TLTP project at the outset. The aim of the TLTP was to get away from the "not invented here syndrome" by bringing academics together in a common project. To a very great extent Mathwise has been successful in this. The project prompted a community of users and developers to come into existence, with friendships developing on the basis of shared trust in an innovative project.
My own approach to Mathwise as a lecturer has been simply to get on and use it. I do not agonise about email being a different form of communication from letters or the telephone, I just get on and use it, although I do recognise that there are significant differences in how you can say things in an email. Even in a Web-based series of articles like the present one, the conventions of expression adopted are perhaps subtly different (more informal) from those which are the norm in academic journals. Mathwise as a medium for assessment is significantly different from other methods and there are significant hurdles to overcome, but its potential benefits are worth the pursuit in the long run.
If there is to be a next stage for Mathwise, then "offspring of Mathwise" will be Web-based. The Mathematics community should carry forward the lessons learned in the Mathwise project and move on into a Web-based version, which I think should look quite different from the CD version. Assessment would need to become more prominent and the place of the text-based narrative sections would need to be re-examined carefully and put into a different format.
I recall having a conversation at a conference with Mike McCabe and Cliff Beevers a few years ago. It was in the days when the Web was the latest new thing and we were, somewhat sadly, bewailing the fact that CD materials like Mathwise were going to struggle. We felt like children on the beach who had just built a magnificent sandcastle, only to find that the tide was coming in and would shortly swamp it. "What will be left after the wave has come in and swamped our sandcastle?" we asked. "The people who dared build it in the first place!" replied Cliff.
Acknowledgement
I am very grateful to the Leverhulme Trust for its funding, allowing me to continue research into the use of Mathwise during the session 2001-2. This has enabled me, among other things, to conduct class observations and student focus groups, so that the observations and conclusions given in this article are based on actual observation of students using Mathwise. It is intended to write up the research in the form of academic papers in due course.
References
[1] Harding, R and Quinney, D (1996), Mathwise and the UK Mathematics Courseware Consortium, Active Learning 4, 53.
[2] Pitcher, N (1998), Educational Software in Mathematics: Developing and Using a Mathwise Module, International Journal of Mathematical Education in Science & Technology 29 No. 5, 709-720.
[3] Beevers C E, Bishop P and Quinney D (1998), Mathwise Diagnostic Testing and Assessment, CITech 191-205.
[4] Scott, T D and Pitcher, N (1997), SUMSMAN: Scottish Universities Mathematics and Statistics across the Metropolitan Area Networks, Proc. Third International Conference on Technology in Mathematics Teaching, Koblenz, CD-ROM ISBN 3-00-002330-5.
[5] Pitcher N, Goldfinch J and Beevers C (2002), Aspects of Computer-Based Assessment in Mathematics, Active Learning in Higher Education, Vol 3(2): 159-176.
[6] Beevers C E, Youngson M A, McGuire G R, Wild D G and Fiddes D J, Issues of Partial Credit in Mathematical assessment by Computer, ALT-Journal 7:26-32
[7] Greenhow M (2002), Answer Files – What More Do They Reveal? Maths CAA Series
[8] Davidson K M (1997), Statwise: Multimedia Statistics Courseware, Maths&Stats, CTI Newsletter, 8, 3, 15-18