Maths CAA Series: November 2002

Calmaeth, automated diagnostics in Calculus, Statistics and Linear
Algebra - some comments by an outsider

Dr Nathan Scott, The University of Western Australia
Reporting on the work of Dr Kevin Judd, also of UWA
email: nscott@mech.uwa.edu.au

Index to article

	Abstract
	1. Introduction
	2. Guest Account and sample screens
	3. Improvement Path
	4. New Packages and User Experience
	5. Conclusion

 

Abstract

In 1995 Prof. Brian Stone of UWA and I had a simple network-based tutorial system set up for Engineering Dynamics. The facility used by the students was a laboratory of 128 Macintosh LC computers which was jointly owned with the Mathematics Department. As a result the Mathematics Department saw what we were doing and thought perhaps it might work for first-year Calculus. Kevin was asked to generate some problem sets for Calculus. Being a true mathematician Kevin was not satisfied with the crude approach used in our problem sets, and instead he began to think more generally about how to offer a student diagnostic feedback for errors.

1. Introduction

The first version of his tutorial system (Calmaeth) was ready by July 1995, our second semester. Initially the delivery platform was the Macintosh printer network and HyperCard, with a custom central server process - the Web had not quite been invented! The heart of the system was and is Mathematica(TM). Kevin wrote a set of packages which:

1) Could generate random questions for students based on patterns e.g. for the derivatives topic it could give each student a slightly different initial expression.
2) Could adapt student input so that it was ready for Mathematica to process (this is harder than it sounds as Mathematica is very strict about notation, closed brackets etc., whereas students are not).
3) Most importantly, could recursively apply a set of rules to the student answers to generate detailed, helpful English feedback.

2. Guest Account and sample screens

Kevin has set up a guest account on his system and this is probably the best way to see what he has done:

http://calmaeth.maths.uwa.edu.au/
Click the Guest Login button.

For your convenience I have also captured some screen shots of the system in action, and have added a few annotations:
http://www.mech.uwa.edu.au/nws/ae/calmaeth/

3. Improvement Path

In 1996 Kevin paid one of his postgraduates (Ronald Monson) to improve the diagnostic packages. Ronald examined the system logs which had recorded all the student answers and system responses, concentrating on student input which baffled the diagnostic packages. He was able to add further tests to the packages such that 90% of all student answers now produce a meaningful response. Only once in ten submissions does the system say, "sorry, I cannot work out what you have done" and in those cases the input is usually non-mathematical or very odd indeed, and human help is probably needed.

Calmaeth has been used heavily since 1995, with up to 1000 students at UWA using it each year. It has also been used at several other Universities although I don't know which ones or whether they are still using it. The Mathematics department are pleased with it because it has allowed them to run effective problem classes in first year subjects more cheaply. Some human help is available in the computer lab during certain hours, but only at the level of 1 staff member for 50 or more students - which has proven to be enough.

4. New Packages and User Experience

Since 1995 new packages have been written for the subjects Linear Algebra and Statistics. Kevin has also created systems to assist staff in writing new problem sets and these have been used by two or three staff in his department. Multi-part problems are possible and are used extensively in the Statistics problem set. The system went "web based" in 1997, the current server is a set of PERL scripts running under LINUX.

In small group problem classes - the traditional kind - the human tutors do not have the time or the patience to enforce complete accuracy and as a result tend to "check through" answers that look reasonable. Calmaeth is not so lenient and students must get every part of every answer correct. There is some evidence, I hear, that students who have used Calmaeth go on to become honours students who make fewer low-level calculation errors.

Many of the first-year students who use Calmaeth have experienced difficulty in using the diagnostic messages, because the messages use many words such as "numerator", "exponent" and so on. It seems that for many students these words are not yet second nature and are a barrier to understanding the mechanics of the maths operations. An improvement to the system that Kevin has spoken about, but has not yet implemented, would be to have much more graphical feedback such that errors in an answer are highlighted in colour, for example.

5. Conclusion

Kevin's diagnostic packages have changed the way maths teaching is done in the early years at UWA, and I think the change can be seen as positive both financially and educationally. I hope you will try them for yourself. I am not an expert in this area but I feel Kevin may have solved a problem in 1995 that many others are still treating as unsolved.