Maths CAA Series: August 2002
Page Guide: Home > Articles > maths-caa-series > |
Abstract |
|
1. Introduction |
|
2. Background |
|
3. The do-it-yourself CAA Solution |
|
4. Student Reaction |
|
5. Conclusions |
|
References |
Abstract This article describes an example of what can be achieved by a very simple approach to Computer-Aided Assessment in a first-year mathematics module. The reasons for this approach are discussed, together with the extent to which it succeeded, the problems which arose, and the students' reaction to it.
1. Introduction
The current level of interest in Computer-Aided Assessment (CAA) in Mathematics and Statistics is shown by the existence of this monthly series, and the number of articles concerned with it in other LTSN publications recently [1-3]. The approach taken depends very much on the resources available - either academic and programming time to produce a suitable system, or technical support for a system acquired from outside the institution. This article discusses what can be achieved with a relatively small amount of time to produce home-grown CAA software.
2. Background
The use of computer-aided learning in general, and the Blackboard system [4] in particular, is currently being promoted in Sheffield Hallam University. A project in the School of Computing and Management Sciences has evaluated CAA in three different modules in order to build up experience in the school. Two modules were about databases, and the module leaders chose to ask multiple-choice questions using Blackboard. I was concerned with a first-year mathematics module, and chose to write a simple program to provide more flexibility.
The Mathematics for Computing module is taken by about 300 computing students in their first year. Their mathematical knowledge and ability is very varied, but the majority have not taken A-level Mathematics. There are problems in motivating the students, many of whom do not like the subject; the carrot is that mathematics has many uses in computing, but in case that fails to motivate them the stick is frequent assessment to encourage attendance. Frequent assessment of a large group of students obviously causes problems with marking; the approach taken has been to assess a very small piece of work (one or two simple questions) each week, and to give a few multiple-choice assignments and tests, to be marked by an optical mark reader (OMR), as well as conventional written assignments.
Experience with processing multiple-choice answers using an OMR has been generally good, in spite of the obvious limitations on the type of question which can be asked. Its use is flexible from the students' point of view because the questions are provided on paper so that they can work on them anywhere. There have, however, been some local technical difficulties. Hallam students have an eight-digit student identifier which is used on their answer sheets, but on the same card they also have an eight-digit library number, which exacerbates the inevitable problem of omitted or incorrect identifiers. Also, the software which processes the OMR output allows only five choices per question, which the module team consider to be insufficient, so a modified answer sheet and extra software is needed to allow up to ten choices.
Plagiarism is a potential problem with any assessed work not produced under examination conditions. We encourage students to discuss their work with each other, but emphasise that for assessed work any discussion should stop at an early stage so that what is handed in is an individual effort. Some students carry collaborative working too far, and produce answers where it is clear that the main ideas have been worked out in a group even though they have been individually written up; a few simply copy from other students. Detecting these cases and distinguishing between different levels of collaboration is possible for written assignments submitted by a small student group, but with large numbers, particularly when more than one tutor is doing the marking, it can easily be missed. An assignment which uses multiple-choice questions does not leave any clues to plagiarism in the wording used in answers, but does allow some suspicious cases to be easily identified, namely where two or more students give identical incorrect answers to a large number of questions.
Some types of question, particularly those which involve numerical calculations, can be altered to allow students to have individual variations. This does not eliminate the possibility of students getting other people to do their work, but it does make it harder for them, as students who allow others to copy their work may not be willing to do the calculations again for them. Students may, of course, give each other help such as "To answer Part (b), you subtract the second number in the question from 1, then multiply by the first and second numbers, then take the square root". There is at least a possibility that both students involved may learn something from this process. However, it would be hard to detect, either when the answers are submitted on paper or when they are processed electronically.
3. The do-it-yourself CAA Solution
To cater for more flexible answer types, to allow individual numerical values in questions, and to overcome the difficulty with incorrect student identifiers, it was decided to use an interactive system. The assessment system provided in Blackboard was not suitable for a number of reasons: numerical answers are not catered for in the current version; mathematical text can be put into questions only as screen images, resulting in poor quality; and it is not easy to print the questions onto paper to be worked on away from the computer. Rather than use an existing alternative system, without local technical support, it seemed easier to develop a program, using Borland C++ Builder.
The system was used for an assignment on elementary probability and statistics in May 2001. A random number generator was used, with the student identifier as seed, to provide questions with (usually) different numerical answers for different students. The questions could be viewed one by one on the screen, and saved to a file for input to a word processor and printing. Printing the question sheet was encouraged, because it was felt that students would then put more time and thought into their answers than if they were working everything out while using the computer. Students were required to input their name and identifier, and to choose a password. The student's answers were saved to a file, which could then be reloaded into the program, as often as required, for answers to be modified. The program checked the name and identifier against a stored list, and checked the identifier and password when a file was reloaded. The answer file was encrypted, though not strongly. Students submitted their answers by sending their answer file as an email attachment.
A week after the submission deadline, students were sent a modified version of the assignment program. This program read their answer file and gave the correct answers, with an indication of the method of calculation, as well as their marks. A similar program was run to read all students' answers and produce a file of marks. (See Fig 1)
Fig 1 A question with answers, as given in feedback to students
I developed the programs as time allowed over a few months; the total time was approximately equivalent to two weeks' full-time work. I had worked as a programmer in the 1980s, but had had very little experience of programming using a modern graphical user interface; someone already familiar with the necessary programming techniques might have taken slightly less time.
The aim was to have a simple system, using a stand-alone program in a single file which could be distributed as an email attachment. This involved building the questions and list of student identifiers and names into the executable file. This ease of use was nearly achieved, but it was found at the last minute that the program needed a dynamic linked library (DLL) which had been installed on the development machine and machines in nearby student labs, but not necessarily on students' own machines. A copy of this DLL file was attached to the email sent to students, with instructions on how to install it if necessary.
Interfaces with other software were kept to a minimum in the assignment program; for example, it did not directly print the student's question file or start a word processor, and did not generate an email to send the answer file. This was to avoid technical difficulties which I might not have been competent to solve. This policy was largely successful, in that students had little difficulty in using the other systems to save and submit their answers, although a few students initially claimed not to know how to download and run an email attachment. One bug in the program caused some difficulty, however. When an attempt was made to save the answer file to a disc with no space available, it actually saved a file with size zero, without giving an error message. This happened to a number of students who had filled their allocated space on the network server, and some of them submitted the empty file without noticing that the size was quoted as zero in the email message.
From the point of view of the administrative work needed to do the marking, the assignment went reasonably well. It was necessary to save manually each student's email attachment to a folder where they could all be read by the marking program, but this tedious job was not too time-consuming. The answer file was named by default using the student's identifier, but a few students, rather unhelpfully, changed the name to something like "answers"; the identifier was, however, also stored in the file. Some students submitted their file of questions, or even the program, instead of the answer file. These mistakes all contributed to the time needed to administer the assignment.
It will be reasonably easy to modify the assignment program to allow it to submit answers by sending an email itself, provided that the student is logged into an email system; however, this extra complication might introduce a greater need to sort out technical difficulties in use. The marking program could read the tutor's mailbox and immediately extract the necessary information and send an email to acknowledge receipt of the answers.
The particular subject being assessed did not need questions containing anything other than plain ASCII text, and lack of time prevented their format being improved. Thus the current system has not overcome this limitation in Blackboard. However, it would not be difficult to incorporate questions which use Rich Text Format [5].
4. Student Reaction
Students were asked for feedback on the assignment by including a short questionnaire, with replies included in the answer file. Of the 258 students who submitted answers, 200 answered the questionnaire. Exactly half the respondents had used a PC in their own or someone else's home, and half in the university or in more than one place. Asked where they worked out the answers, 78% of respondents did it mainly on paper away from the PC. More students were satisfied with the method of submitting their work than not; 42% found using the program to submit answers more convenient than a paper multiple choice assignment, 34% found it less convenient, and the rest found it equally good. Compared with writing answers on paper for manual marking, 46% found the program more convenient and 29% less convenient.
A few students sent additional comments. One or two said things like "The program is rubbish" without giving further details. A few were concerned about not getting credit for the method of calculation, or about getting no marks at all for an answer which differs from the correct one by no more than rounding errors, though the latter worry was due to a misunderstanding of how the marking was done. One student mentioned the convenience of submitting work at the last minute without having to come in to the university to join the queue with other students in the same situation, and approved of the different numbers in different students' questions to prevent cheating.
5. Conclusions
This article has described an example of what can be achieved by a very simple approach to CAA using a limited amount of time, and has concentrated on technical and administrative issues more than on pedagogy. The software which has been produced can easily be improved to overcome some of the problems which have been found. However, to modify it to make it more generally useful would take considerably more effort, and might destroy the simplicity which was one of the criteria used in designing it.
References
[1] N. Strickland. Alice Interactive Mathematics. MSOR Connections, 2(1), 27 - 30 (February 2002).
[3] Workshop report: Assessment for a Purpose. MSOR Connections, 2(3), (August 2002).
[4] Blackboard Inc. Welcome to Blackboard <http://www.blackboard.com/> (accessed 9 July 2002)
[5] CG Information. Rich Text Format (RTF) Version 1.5 Specification <http://www.biblioscape.com/rtf15_spec.htm> (accessed 9 July 2002)
