Maths CAA Series: Nov 2001
Page Guide: Home > Articles > maths-caa-series >
|
1. Introduction |
|
2. Background |
|
3. Diagnostic Testing |
|
4. Computer based support material |
|
5. A Case Study 2000-2001 |
|
6. Conclusion |
|
7. References |
For students entering Higher Education courses in Science and Engineering there is always some level of pre-requisite assumption and reliance on prior knowledge in a range of topic areas and mathematical skills. (See for example Kitchen (1996), Hirst (1997), and Lawson (1997).) Such courses also tend to recruit large numbers of students with a rich diversity of intake qualifications and prior experiences. The need to assess on entry the current active ability of students to any course is crucial. To be able to do so rapidly and effectively, and provide suitable student centred support, remains an ongoing challenge. In this paper we will describe one possible approach that uses technology for diagnostic testing and follow up support.
Over the last decade the nature and background of the students who arrive at our universities each September has changed markedly. The structure of a modular A-level curriculum, in particular Curriculum 2000, means that they have had a considerable range of mathematical experience. (See Porkess (2001)). Into this melting pot we need to throw mature students returning to higher education, students entering with GNVQs, and foreign access students. The situation is further complicated by the increasing participation rates of 18 year olds. As a consequence it is fitting to ask:-
"Do we really know our students?"
Clifford (1993) conducted an experiment to compare the actual mathematical skills and ability of engineering students arriving at a university with the skills and ability that staff expect them to have. The results showed a wide discrepancy. Recently a similar experiment was carried out at "another" university. Students were asked 5 questions (Note 1) that were typical of the material we might expect a reasonable A-level student to know. They ranged from solving quadratic equations, simplifying expressions to evaluating a simple definite integral. The results are shown in Table 2.1.
| Expected | Achieved | |
| Question 1 | 92 | 67 |
| Question 2 | 88 | 58 |
| Question 3 | 64 | 67 |
| Question 4 | 63 | 100 |
| Question 5 | 81 | 67 |
We leave the reader to draw their own conclusions but suggest that they might like to carry out a similar experiment!
In Hibberd, Looms & Quinney (2001) the need to establish an educational strategy that begins with diagnostic testing, provides suitable support material which is a symbiotic relationship between computer based material and tutorial support, was considered.
For many reasons, the need to provide suitable diagnostic testing is taken for granted. Within the mathematics fraternity it is apparent that if every institution were to develop independently its own tests then this may be both wasteful and inefficient. In 1996 the Heads of Departments of Mathematical Sciences in the UK (HoDoMS) funded a WWW site giving information, contacts and case studies of existing diagnostic tests. http://www.keele.ac.uk/depts/ma/diagnostic/. This site contains links to the diagnostic tests used at a number of universities and a selection of case studies which give details of how diagnostic testing is carried out and, just as importantly, how students are supported thereafter.
The need to rapidly assess the current ability of students on entry to any course is two-fold:
(i) Provide teaching and tutorial staff with a global assessment of the current active ability of each student on a chosen range of topics. Quantitative information attained is valuable in the following ways:
(ii) Provide students with useful individual feedback before problems escalate. Benefits to students include:
Multiple Choice Questions (MCQs) are attractive to those looking for a way of assessing students arising from their ease of marking by providing a computer based form of assessment. Further details are given by Brydges & Hibberd (1994) and Beevers, Bishop & Quinney (1998). At Keele University we have used a MCQ diagnostic test for a number of years in order to identify any students who may need particular attention. The test consists of 20 MCQs selected randomly from a bank of about 50 questions. A typical question is shown below.
The aim of the test is not simply to return a numerical mark but identify skills that might be lacking. To this end the student's responses are analysed to determine the student's capabilities in 10 distinct skills; at the end of the test they are presented with a diagnostic screen as shown in figure 3.2. It should be noted that the test gives partial credit by grading the skills that might lead a student to select one of the incorrect answers and rewarding them accordingly. The student can decide to abstain from a question; in which case they are not penalised for selecting a wrong answer. However, such a decision indicates a deficiency of a particular skill and this is reflected in the final diagnostic report. The final score given to the student may not be the number they answer correctly.
At Keele University, we ask all our 1st year mathematicians to take this test in their first week. The results are then available when the student meets their departmental tutor and they can discuss what action needs to be taken. In this way both the student and tutor are able to discuss the student's skills and ability based on the evidence at hand and requires minimal effort on the part of the staff; their skill can be used to analyse the diagnostic report and advise the student on their individual needs.
After the diagnostic test all information is collated automatically. This includes individual student performance and the way the whole cohort performed on individual questions. Examination of the raw data files provide very detailed information on total student performances that can be readily used as a reference for module revisions or for directing supplementary resources. However, it is necessary to show that the diagnostic test provides a realistic indicator of individual students' capabilities. To do this, during the academic year 2000-2001, students were asked to take both the diagnostic test and a written paper and the results compared. A total of 87 students entered Principal Mathematics; all students took both the diagnostic test and also completed an exercise that involved a large number of problems involving differentiation at various levels of difficulty. A statistical comparison of the written and diagnostic test showed that the scores are highly correlated (r=0.75, p<0.001) and that a simple linear regression model accounts for 55% of the variation of the marks. A scatter plot, together with regression line is shown in figure 3.3. We conclude that the diagnostic test is a good predictor of an individual student's skills in differentiation. This is significant, as the reduction in workload required in using the automation provided by the CBL diagnostic test can be significant.
A diagnostic test described above has been operating in the Mathematics Department at Keele University since 1995/96 and figure 3.4 illustrates results of profile skills for the whole student cohort in five successive years. The wide discrepancy, year by year, indicates that simply providing common remedial courses will not be suitable. Indeed the variation over time suggests that a different set of support material would ideally be needed each year. It seems appropriate, therefore, to look at the microscopic scale and try to focus on individual students and attempt to assign each student suitable support material. Providing individualised programmes of study using computer based self-study programmes based on the results of the diagnostic test, as shown in figure 3.5 may be a solution to this problem.
Although our primary aim of this work is to help students to achieve their potential in mathematics, the gathering of information regarding the mathematical abilities of students in higher education is also possible. Such information will help long term planning in higher education and also provide positive feedback to secondary education in general.
The results of the diagnostic test between 1996 & 2000 were sufficiently encouraging that it was decided to integrate the process diagnosis-support into the first year programme. Follow the diagnostic test students were allocated assorted computer based teaching modules in the Differential Calculus. After completion students were expected to take the associated test and the recorded score was incorporated into their module grade. The additional overheads in students submitting work on a variety of modules was minimal since the diagnostics marks, required module and assessment were all collated automatically. The response from students has been exceptionally positive, in that the students have requested similar material to extend the diagnostic process to consider integration in more detail.
Gains made from the implementation of diagnostic testing are limited without also providing suitable "support material". Generally, provision is sought which can be used to provide an effective way of ensuring all students have individual access to the required core knowledge and appropriate support tailored to their individual needs. This can be accomplished through tutors, clinics, supplementary lectures, and mathematics resource centres, etc. (Lawson, Halpin and Croft, (2001)) However, experience has shown that even though the weaknesses of individual students can be detected using diagnostic testing the restrictions of individual and teaching timetables make it difficult to allot specific times when students can be supervised to ensure that any remedial work is carried out. Therefore, it seems appropriate to use CBL to provide flexible mathematics courseware support.
During 1996-1999 the mathematics department at Keele University pioneered the use of the TLTP material, Mathwise, to provide individual study profiles which were automatically allocated following the diagnostic test. See figure 3.5. (Hibberd, Looms & Quinney (2001)). However, many students are becoming familiar with computer algebra systems (CAS) such as Mathematica, Maple, Derive, etc. Although these systems are excellent at "Doing" mathematics they leave much to be desired for teaching and learning mathematics. To this end we have been investigating the use of a CAS system that concentrates on teaching and learning, and how such a system can be integrated to provide the student support needed to follow up a diagnostic test.
A new software package called Calculus Machina is being developed, which has been designed not only to be capable of CAS but also of revealing the steps which are required to evaluate derivatives and integrals. The interface allows students to type in their own expressions and see them displayed immediately in a "pretty print" form after selecting the relevant expression. Alternatively, mathematical expressions can be entered using simple templates. (See figure 4.1.)
Once a function has been defined the software will either display the steps required to determine the derivative, as shown in figure 4.2. In this example, the Calculus Machina has been asked to differentiate sin(x2). Notice that it recognises that it is necessary to use the Chain Rule (flagged by the text Derivative of Composite Function) and then reveals the steps needed to continue. Theses flags also provide a hypertext link to context sensitive help that allow the student to "drill down" and gain additional help as shown in figure 4.3. These pages are derived from Hughes-Hallett, et al (2002) but an alternative version is linked to Anton, Bevins and Davis (2002). Future versions of the software will enable an instructor to add links to alternative texts and additional material. The advantage with Calculus Machina is the ability for the students to type in their own problems or for it to generate practise problems for the student to attempt to re-enforce their skills in this topic.
Since Calculus Machina is able to differentiate almost all functions met in first and second year mathematics and documents all the steps involved, it might be thought that this will encourage students to take a very passive role and allow the computer to do the work. However, Calculus Machina has a second, more educational, mode in which the student has to take a much more active part in the process. This mode, called Udo, is illustrated in figure 4.4. Once again Calculus Machina has been asked to differentiate sin(x2) but now the student has to supply the requisite substitution which is then checked before they are permitted to proceed. In this mode Calculus Machina can play the part of an individual tutor checking on each step and allowing students as much practise, as they need. The software includes the ability to generate further problems which are closely related to the current problem to give further practice.
To investigate the effectiveness of the Calculus Machina, the students studying Principal Mathematics at Keele University during the academic year 2000-2001, were divided into two groups. Those scoring in excess of 65%, on the diagnostic test, (Group A) were asked to look at a Mathwise Module called Applications of Mathematics. The remaining students were further randomly sub-divided into two further groups (B1 and B2). Group B1 was asked to study a Mathwise Module: Rules of Differentiation and Group B2 was asked to use Calculus Machina. The aim of the project was to compare the performance of groups B1 and B2. To do this Groups B1 and B2 were asked to retake the diagnostic test at the end of their study and also complete a paper-based questionnaire.
A total of 28 students completed the pre and post-diagnostic test though somewhat fewer also completed questionnaire. The students in Group B1 had a mean baseline score of 49.53 whilst those in Group B1 scored slightly less, 43.3 though this difference was not significant, (p=0.23 using a t-test). 2 students in Group B2 were not included in the analysis, as they would have skewed the result even further in favour of the Calculus Machina. (One student scored 1 and 68 the other 8 and 40, and it was felt that these would skew the data in both absolute and relative terms.) To investigate the effectiveness of the packages allocated to the two groups the mean paired absolute differences of the two groups were analysed.
The results of this trial are given in Table 5.1, and suggest that Group B2 have improved significantly much more that B1 (p=0.005) even though their pre-test score was slightly poorer. Analysing the relative improvement in diagnostic score after using the software gives a similar result. Even though there is substantial variation in the results observed and the sample sizes are relatively small we can conclude that, based on these results, the Calculus Machina appears to be the more effective software when used in this context.
| Group | Number | Software | Pre-test score | SD | Mean Difference | SD |
| B1 | 13 | Mathwise | 49.53 | 14.61 | 5.38 | 10.39 |
| B2 | 13 | Machina | 43.30 | 10.94 | 22.4 | 17.02 |
It must be noted that a direct comparison between the Calculus Machina and Mathwise: Rules Of Differentiation is a little unfair as they are several generations of software apart and the Calculus Machina is designed specifically for the Calculus whereas Mathwise covers a much wider remit. Nevertheless, the mathematics department at Keele University has invested substantially in its use of Mathwise and there is substantial inertia in changing to a new system. A similar experiment was conducted during the academic year 2001-2002 and the results were very similar. The major advantage of the Calculus Machina is its ability to accept problems entered by the student and disclose and document how the derivative or integral is found.
18 completed questionnaires were returned; 9 from Group B1 and 9 from Group B2. Respondents reported a wide range of reasons for studying Mathematics or Statistics and a wide variety of topics in which they had perceived strengths and weaknesses. Most of the students (12) regarded the diagnostic test as accurate.
Students varied widely in their attitudes to computer use and their views on computer-based learning. Some appreciated the fact that the computer allows them to work at their own pace, provides instant feedback, and leads them step-by-step through methods; others found the experience somewhat stressful. 11 of the respondents found the classroom sessions more useful than the lab sessions, (due to the perceived slowness and difficulties in inputting mathematical expressions), 3 preferred the lab sessions, and 4 felt both were useful in combination.
The Mathematics Department has used a simple diagnostic test for some years as a means of differentiated teaching and support for students. This project has now verified that the use of the diagnostic test provides a simple cost-effective means of identifying student weaknesses in differentiation.
The mathematics department at Keele University has made use of several modules from Mathwise over the last 5 years but the capabilities of more recent software, in the case of the study we have carried, Calculus Machina, has been shown to be more effective. Accordingly we aim to build it into the week that the Department has set aside for developing the students' skills in Introductory Calculus from the academic year 2002-2003.
Existing technology and courseware is available to help detect areas of mathematical weakness at individual student level. Although discussions with course tutorial support staff are vital, the computer-based profiles provide a pro-active mechanism for the early identification of student weaknesses. The basis of this paradigm is dependent on the development of study skills by individual students and the inclusion of both summative and formative assessment can help re-enforce this. The same software can also be used to gather information on the cohort as a whole and also to track the performance of students on a year-by-year basis.
Anton H, Bevins, and Davis (2002). Calculus, 6th Edition. John Wiley and Sons Inc, 0-471-15306-0
Beevers C. E., Foster M. G., McGuire G. R. and Renshaw S. H. (1992) Some Problems of Mathematical Cal. Int. J. Comp Educ. 18 pp. 119-125.
Beevers C. E., Maciocia A, Prince A.R., and Scott T.D. (1996) Pooling Mathematical Resources Active Learning 5 pp 41-42.
Beevers C.E., Bishop P, and Quinney D.A. (1998) Mathwise, diagnostic testing and Assessment, Information Services & Uses, Volume 1, 1-15.
Brydges S & Hibberd S, (1994) Construction and Implementation of a Computer-Based Diagnostic Test. CTI Maths and Stats. 5/3 pp9-13.
Clifford H (1993) A comparison of expectations and achievement in a prior knowledge assessment of B.Eng. (Hons) course students. International Journal of Mechanical Engineering Educations, Vol 22, Number 1. pp55-68.
Hibberd S, (1996) The Mathematical Assessment of Students Entering University Engineering Courses. Studies in Educational Evaluation 22 pp 375-384.
Hibberd S, Looms A. & Quinney D.A.. (2001) Computer based diagnostic testing and support in mathematics, To appear in Innovative Teaching Ideas in Mathematics, Ed Mohammad H. Ahmadi.
Hughes-Hallett, D, Gleason A, McCallum et al. (2002) Calculus. 3rd Edition, John Wiley and Sons Inc, 0-471-16442-9
Hirst K. (1997) Changes in A-Level mathematics from 1996. University of Southampton.
Kitchen A. (1996) A-Level Mathematics isn't what it used to be; or is it? Mathematics Today. 32/5 pp. 87-90.
Lawson D (1967) What can we expect of A-level mathematics students? Teaching Mathematics and its Applications 16/4. pp 151-156.
Lawson D, Halpin M & Croft A, (2001) After the diagnostic test - what next? Evaluating and Enhancing the Effectiveness of Mathematics Support Centres, MSOR Connections, Volume 1, Number 3, pp 19-24.
Porkess R, (2001) Mathematics in Curriculum 2000: What will students know? What will students not know? MSOR Connections, Volume 1, Number 3, pp 35-39.
1. This was based on a survey carried out by Duncan Lawson,