Maths CAA Series: March 2003

Coping with diversity in recruitment. Can CAA help?

Index to article

	Abstract
	1. Introduction
	2. Module content and assessment
	3. The Tests
	4. The students
	5. Results and Discussion
	6. Further Comments

 

Abstract

Many Engineering awards recruit students from diverse academic backgrounds that include A-level, BTEC qualifications, Foundation Years and overseas qualifications. In addition to this there will be mature students returning to education after an absence of more than five years. One of the main areas of weakness possessed by this diverse population is mathematics and many of the students require a considerable level of support if they are to progress beyond the first year of their award. CAA appears to offer certain advantages for this type of course. Students receive instant feedback on their progress, and they can easily repeat the assessment, potentially linking the assessment directly to learning. This exercise can be carried out efficiently for large numbers of students and where tutor support is required, it is being given to a student already informed, to a certain extent, on where their weaknesses lie. In this article the use of CAA on a module with the above student profile is discussed and its effectiveness considered by tracking the performance of a group of students on first and second level mathematics modules. The results are mixed, as may be expected for this group. However, when viewed over the longer term, the majority of students from each of these different cohorts did make significant progress in the subject.

1. Introduction

The decline in the number of students taking A-level mathematics has meant that many engineering departments recruit a substantial proportion of their students from a non A-level background. The majority of UK students entering an engineering award without A-level mathematics will have either taken a foundation year, a National Diploma or a Higher National Certificate in an engineering discipline. Although the students from these various courses will have studied a similar curriculum to an A-level mathematics student, it would be misleading to describe the experiences of these students as equivalent to each other. Despite the well-documented weaknesses of students with A-level mathematics, such students will usually be more confident in their algebraic manipulation, have a wider knowledge of mathematical functions and will have applied the methods of calculus to a more difficult set of problems. However, while the experiences of these different groups are not strictly equivalent, students without A-level mathematics can be successful. It follows that many lecturers delivering first year engineering mathematics courses are teaching a group that is certainly mixed in ability, but also mixed in maturity and experience.

In order to cope with the diversity that is inherent in recruiting students from different academic backgrounds, it is necessary to create sufficient time for students to address their weaknesses. Although this is an obvious point to make, where it leads is that the course delivery must include an element of self-paced learning. The pace at which material must be learnt is clearly influenced by the assessment of the module. Computer aided assessment is interesting in that it opens up opportunities to set up a very flexible assessment regime. For example, students may retake assessment with little additional cost to the tutor, creating a potential learning cycle of assessment, reflection and action. When used in this way, computer aided assessment provides a method of self-assessment for the student. The course must then provide sufficient time for the student to act on this information. Where the problems that have been identified are minor and easily rectified, subsequent support may be offered through a variety of learning materials. However, where the nature of the problem is more severe then it is unlikely that a student would be able to remedy the problem without tutor support. In such a case the feedback received by the student may do little more than indicate that there is a problem to be addressed without accurately identifying its nature. The success of the overall learning support strategy depends on the nature of the help provided by the tutor. One of the key contributions made by computer aided assessment to this strategy is that the tutor can build upon work already carried out by the student.

While it may be clear that computer aided assessment can initiate learning activity, it is important to ask whether this activity makes any difference to a student's performance at the end of the year and in subsequent years? Generally speaking, computer aided tests tend to be short questions, either multiple choice or those that require a numerical answer. Algebraic manipulation skills can only be inferred from the student response. It may be possible through carefully constructed feedback, such as that used by the Mathletics suite of questions, developed by Martin Greenhow at Brunel University, to get closer to the possible source of any difficulties. More sophisticated test structures are of course possible and the interested reader is directed to several of the other articles in this CAA series. However, written examinations at the end of module and the actual use of mathematics in the wider context of an award will require skills that are not being assessed by the computer based tests.

In this investigation the performance of a first year engineering group is tracked into the second year. The period between starting the first year engineering mathematics course and sitting the end of module examination for the second year engineering mathematics course is just over fifteen months in length. A comparison is made between those who came onto the course with an 'A' level, Foundation Year or B-TEC mathematics background. On both first and second year engineering mathematics modules, the students take computer-based tests as part of their assessment. They then have to sit an end of module written examination. There is no other assessment on the module. The computer-aided assessment is designed to encourage students to work regularly throughout the course. It is therefore of interest to establish whether the strategy adopted at the University of the West of England does indeed promote effective learning for this diverse population of students.

2. Module content and assessment

2.1 Engineering Mathematics 1

The first year engineering mathematics course covers the topics of algebraic manipulation, functions, complex numbers, calculus and linear algebra. In the year under consideration, the course also contained some material on probability distributions. This topic has now been removed. The module, which is core for all BEng awards within the faculty and the BSc Music Systems Engineering, runs throughout the academic year.

The course assumes a working knowledge slightly beyond AS level mathematics. The assessment is organised into two components. The end of year examination is worth 70% of the final mark, the computer based tests make up the other 30%. The CAA component consists of four equally weighted tests on Algebra and Functions, Calculus, Linear Differential Equations and Linear Algebra. Each test consists of about twelve questions. Students have three attempts at each test to be taken over a period of two and a half weeks. The period is chosen to allow all students including the part-time students to have access to tutors during the test period. Questions are randomly drawn from a central bank so students take a different set of questions with each attempt. The highest scoring attempt is recorded as the mark for that test. The tests are managed using the web based Questionmark Perception software. Being able to gain remote access the tests is vital for part-time students who only attend the University on one day each week and do not necessarily live nearby.

To pass the module, students must achieve an overall mark of 40% subject to a mark of at least 35% on each component of assessment.

2.2 Engineering Mathematics 2

The second year module is only worth 10 credits and runs during the first semester. The topics covered are Laplace transforms, Fourier series analysis, vector geometry and eigenvalues. Most of the students taking engineering mathematics 1 must also take engineering mathematics 2 as core to their award. The assessment regime is essentially the same as in engineering mathematics 1, but there are only two computer-based tests and the weighting is examination (80%), computer-based tests (20%).

Those students who wish to, mainly from the electronic and music systems awards, may take an additional mathematics option on signal processing in the second semester.

3. The Tests

The Questionmark tests taken by this cohort may be viewed via the link

http://www.cems.uwe.ac.uk/qm/perception.dll

using the login name: "em1access" and the password: "visitor"

4. The students

In analysing the performance of the students a distinction is made between full-time and part-time students. Even though the part-time students have a variety of mathematics backgrounds, they all share the characteristics that they are more mature than the full-time students, are usually returning to education after an absence of at least three years and are studying while meeting the demands of work. Also, a classification is made of those retaking the module as a distinct group. All students in the university have an automatic right to a second attempt on any failed module. The proportion of each group on the module is shown in the table below

Classification	Code	Mode of Study	% of sample
A-Level Foundation Year BTEC (level 3) Overseas Part-Time Retaking Module Not Identified	A F B O PT R U	FT FT FT FT PT FT/PT FT/PT	32 24 13 7 7 5 12

Table 1: Classification of students on engineering mathematics 1

It was not possible to classify about 12% of the sample from the information available. Most of the unclassified sample will be from internal transfers from other awards within the faculty and will have either A-level mathematics or Foundation Year experience with possibly additional mathematics studied at undergraduate level.

5. Results and Discussion

The table below shows some summary statistics for the 2001-02 run of the module Engineering Mathematics 1. The data represents raw assessment marks, no moderation of these marks took place during the examination boards.

Students submitting assessed work	End of module exam attempts	Passed module at first attempt (June)	Passed module after referral
166	153	110	14

Table 2. Progression statistics for Engineering Mathematics 1 2001-02

The progression statistics look reasonable for a first year module, especially one with such a diverse intake. However, the average marks tell a different story. 30% of those who passed the module at the first attempt, were marginal passes. That is, they passed with an examination mark of between 35 and 39%. The effect of this is seen in the low median mark of only 40% shown in Figure 1. Interestingly, the examination results for 2000-01 showed a median score of 46% with only 11% of the passes being marginal. So it appears that, the year under consideration, was a weaker intake than before. The average marks for the computer-based tests are much higher than that obtained for the examination and while this is to be expected, given the opportunity to retake the assessment, the effectiveness of the exercise does need to be questioned.

 

Figure 1: Inter-quartile ranges for Engineering Mathematics 1 assessment 2001-02

In Figures 2 and 3, consideration of the students that passed Engineering Mathematics 1 in 2001-02 analyses their performance on Engineering Mathematics 2, which was examined in January 2003. There were 91 students in this sample.

Figure 2: Inter-quartile ranges for Engineering Mathematics 1 (2001-02) and Engineering Mathematics 2 (2002-03) end of module examinations.

Figure 3: Examination results obtained by the 2001-02 intake on Engineering Mathematics 1 and Engineering Mathematics 2.

The improvement in performance from level 1 to level 2 is clearly illustrated by the statistic that on Engineering Mathematics 1, only 25% of student scored higher than 56% on the examination, whereas on Engineering Mathematics 2, 50% scored over 58% on the end of module examination. The improvement is also shown in figure 3 where the majority of students have achieved a higher score on the level 2 examination. Figure 3 also shows that many of the students who were marginal passes on Engineering Mathematics 1 improved their performance sufficiently to become clear passes (i.e. they scored at least 40%). However, the data also shows that the majority of failures on Engineering Mathematics 2 came from those who scored between 35 and 39% on the Engineering Mathematics 1 examination. This is important as it suggests that our margin for moderating the year 1 marks upwards was not great and there clearly would have been the danger of simply adding to the failure on Engineering Mathematics 2 had we done so. However, despite the encouraging data showing the long-term performance of these students, clearly the experience many had on Engineering Mathematics 1 would not have been a positive one.

Figure 4 shows that the low average mark recorded on the examination is principally due to the performance of the Foundation Year and BTEC students. Incidentally, only two out of forty nine A-level students failed the module.

Figure 1 showed that the performance on the calculus test was significantly poorer than on the other three tests. In Figure 5, we see that the relative performance between the different cohorts, follows a similar pattern to that observed in the end of module examination.

 

Figure 4: Inter-quartile ranges for Engineering Mathematics 1 Examination (2001-02)

 

Figure 5: Inter-quartile ranges for Engineering Mathematics 1 Calculus CAA (2001-02).

 

Figure 6: Inter-quartile ranges for examination results of students tracked from year 1 to year 2

Figure 6 considers the relative performance of the three main cohorts on the module. There were 68 students in this sample (41(A), 11(B), 16(F)). Each cohort improved its average performance from year 1 to year 2. The performance of the Foundation Year students is worth noting. The highest mark in the first year examination was 56% from this group. However, 50% of this group scored 55% or higher on the second year examination. It should also be noted that the improvement in the performance Foundation Year is stronger than that for the BTEC students.

Figure 7: Examination results for students tracked from year 1 to year 2

Figure 7 compares the actual examination results for each student. Again, it is clear that most students have improved their examination results. Many students have improved their performance by over 15%.

6. Further Comments

The observation that students performed so much better on engineering mathematics 1 than on engineering mathematics 2 may be explained by the fact that the level 2 module has a narrower syllabus. A large part of the syllabus is concerned with linear algebra and so the students have not been tested to such a large extent on what was collectively their weakest topic, namely calculus. It is also, fairly easy for students to see the relevance of these topics to their engineering studies. The improvement of A level students who scored poorly on the level 1 module, could simply be down to these students being a bit more mature and motivated at the start of their second year. However, this reasoning does not hold for the Foundation Year students who tend to be quite well motivated in their first year. Their results at the end of year 1 did not suggest in anyway that a significant improvement was to take place just six months later and their first year examination results suggested deficiencies in all areas of the syllabus not just calculus.

Despite performing poorly at the end of the first year, the non A-level students did manage to make good progress in the subject. Therefore, the strategy of encouraging students to work regularly throughout the year has been successful. However, the level of performance on the first year module does need to be addressed. It could be concluded that the end of year examination did not give students the chance to show what they had learnt resulting in low marks and this is clearly one area to consider. However, it is believed that the pass-fail boundary was in the correct place and that the actual standard of the examination paper was appropriate.

One modification made for this year's delivery has been to introduce a series of diagnostic tests at the start of the first year module. Students were allowed multiple attempts at each test and had to complete the exercise before the end of the second week of term. The results from this diagnostic exercise were not surprising with considerable areas of weakness identified in topics such as calculus and trigonometry. However, the first two computer aided assessments have shown improvement in both of these areas. The average mark for the calculus test was 6% higher than the corresponding average for the 2001-02 intake.

The results from this investigation have been mixed. However, students from undeniably weak mathematical backgrounds have achieved good results on a second level mathematics module. In many ways, the performance of the non A-level, UK students were as expected. After all, when the first year course started, many of the Foundation Year students will have only studied mathematics beyond GCSE level for one year. That it took a further twelve months before they consolidated their knowledge and showed some confidence in tackling mathematical problems is not surprising. The investigation has shown that an assessment regime based on CAA can work for mixed ability groups. Clearly some effective learning has taken place during the first year. However, the results suggest that many students were simply not ready for the demands of the end of module examination and with respect to this observation it is clearly necessary to review some of the activities that take place on the module.