Therefore, a candidate in a paper-based examination will normally be awarded
3 or 4 out of 5 marks for the following solution:
If only the final answer is
marked, as is often the case in an ICT examination, then the candidate receives
no marks for the answer –160. This can cause significant evidence of a difference
in marks between the same examination taken in different media (see [4]) -
the longer and harder the questions then the greater the difference.
In reference [5] the authors
investigated alternative ways of using an ICT examination to try to overcome
this lack of partial credit. Both methods, described in [5], involved the
concept of using Steps, that is, breaking the question down into smaller parts
that the candidates had to answer. The two methods tried involved compulsory
steps and optional steps. In the first the candidates had to go through all
the steps, while in the second the candidates were given the choice of taking
the steps or not. The marks obtained in either of these tests were comparable
to those obtained with partial credit in a paper-based examination. However,
even if the marks obtained are similar, this does not mean that the candidates
had shown the same skills. In particular, the use of Steps provided the candidate
with the strategy to do a question. This is normally a skill that is sought
to be tested in these examinations.
To determine whether there
are other possible differences in skills shown in the two types of examination,
it is necessary to take a much closer look at the question and marking scheme.
The points at which marks are awarded could be called the learning points
that are being tested in the question. In Question 1 above these are:
a) knowing the area is a definite integral;
b) writing down the function to be integrated;
c) finding the limits;
d) finding the antiderivative; and
e) evaluating the integral. These correspond to the lines of the answer shown
above and the human marker would award a mark for each one of these learning
points that the candidate performs correctly.
This leads to the following
approach for translation of a question into an ICT format. Begin by looking
at the detailed marking scheme for the question and decide which learning
points are to be tested - these should correspond with identifiable marks.
Some of these marks might be awarded for computations (for example finding
the limits of the integral and the formula for the antiderivative in Question
1), while others are awarded for the strategy involved in solving the problem
(for example knowing that the area is obtained by finding an integral in Question
1). If a mark is to be awarded for knowing the strategy, or where one has
to "know how to" or "use the formula for", it is possible
to set this up as a Step in the question. In the automatic assessment system
used at Heriot-Watt University a candidate has the option to obtain this Step
by clicking on a button labelled "STEPS" (see figure 1). If they
do this then they see some information about how to start the question (this
type of step could be called an Information Only Step) together with possibly
some additional Steps that provide shorter question prompts that try to guide
them to the answer to the overall question.
Figure 1: A question with steps available
Figure 2: The same question with steps open
In Question 1 the Information
Only Step may be "to calculate the shaded area, it is necessary to calculate
a definite integral". As this stands this looks as though it is open
to the same type of criticism about the experiment with steps mentioned above.
However there is a difference here in that to compensate for the lack of knowledge
of one of the learning points, full marks may not be awarded when the candidates
use Steps. For some questions only an Information Only Step is given, but
in Question 1 as well as an Information Only Step there is the possibility
of giving other Steps for which candidates have to provide answers. For example
they could be asked for the limits of the definite integral, the function
that is to be integrated and its antiderivative. These are awarded marks
based on the number of learning points that are being tested. This is very
similar to the way that partial credit is awarded by human markers. Therefore,
using Information Only parts and other Steps could provide a suitable way
of testing the same learning points in a question as those that are currently
measured in a paper-based examination. This would also allow for the possibility
of awarding partial credit in an ICT examination.
2. Some further examples
The following two examples
illustrate what happened in practice when using questions of this nature with
a group of Higher Mathematics students in two Scottish schools. Two sets of
students were selected in pairs matched by ability with one set taking the
written form of the questions and the other set taking the ICT version of
the test. Rough working booklets from the ICT examination and candidates'
attempts in the paper-based examination were used to identify any errors that
occurred in candidates' solutions.
Question
2: Find the gradient
of the tangent to the curve with equation y
= x2 - 8x + 14 at the point (6,2).
The marking scheme gave the learning points as:
a) know to differentiate to find the gradient;
b) perform the differentiation;
c) know to substitute the value of the derivative when x = 6;
d) perform the arithmetic. The steps in the ICT version of this question were
therefore:
1.1
To find the gradient
of the tangent you first need to find dy/dx (an information only Step
corresponding to learning point a); and
1.2 If y = x2 - 8x
+ 14
find dy/dx(and
here the students were asked to type in their answer. This corresponds to learning
point b).
Then, the key part asked for the equation of the tangent (learning points
c and d).
So, the good students could
score 4 marks, usually completing this question without recourse to using
Steps. Those that took the steps and received the strategy lost 1 mark, but
could recover by completing the second step for one mark and gather the final
two marks by writing down the equation of the line, thus, securing partial
credit of 3 out of the possible 4 marks.
A comparison was made between
the marks obtained doing the ICT version of the question with these steps
and a paper version of the question in which partial credit was awarded by
a human marker for answers that were not correct. These used statistical
techniques similar to, but slightly more powerful than, those presented in
[5], essentially using mean marks adjusted to take account of student ability.
It transpired that this question produced no significant difference in marks
between these two types of examination. In general, most errors occurred
at the start of the question - usually due to a student not realising that
differentiation was required to tackle this problem. The actual differentiation
and substitution caused little problem.
Question
3: Find the derivative
of y = (x3-1)/x2.
In question 3 the marking scheme gave the learning points as:a)know to divide
out the quotient; b)perform the division; c)differentiate the positive power;
and d)differentiate the negative power. The steps given in the ICT version
of the question were
1.1
To find the derivative
you first need to divide x3-1 by x2 (an information
only Step corresponding to learning point a).
1.2
If (x3-1)/x2 = xp - xq, what is
the value of p? (part of learning point b).
1.3
If (x3-1)/x2
= xp - xq, what is the value of q? (part of learning
point b).
Again, the candidates using
steps and answering all the Steps and the final answer correctly would gain
3 out of 4 marks. However, this time there was significant evidence of a difference
in marks, between those students doing the paper version, receiving partial
credit and scoring more marks than those doing the ICT version. The reason
for this was because in this case most candidates knew that they had to split
this into two terms (so did not use the steps as they knew the strategy at
the start of the question) and could differentiate the x term. However, many of them could not
differentiate 1/x2.
Those doing a paper-based examination gained a lot of partial credit for incorrect
answers since their correct early working was rewarded. Those students in
the ICT examination who did the question without using steps and obtained
the wrong answer were awarded no marks. This accounted for the significant
difference in marks.
Questions 2 and 3 looked similar in their potential for the successful implementation
of Steps in an ICT examination but the results show that these examples did
not behave in the same way in practice.
3. An alternative approach
The result of analysis on Question 2 shows that
in this case Steps were a good way of replicating the partial credit a human
marker would give, but, this was not the case with Question 3. This may look
disappointing from the viewpoint of replication of results by ICT examinations
from paper-based examinations, but, there is a way to align the ICT questions
so that they better fit the learning points of the original question. This
can be done by a more detailed consideration of the learning points. What
happened was that a pre-existing question from a paper-based examination was
changed into an ICT examination question that tried to re-produce the examination
of the same learning points as the paper-based question. However, if instead
the learning points to be tested were the starting point, then it may be possible
to construct one or more questions that test these learning points equally
well in both paper-based examinations and ICT examinations. Rather than testing
the learning points in one question it is possible to test these separately
as in the following two questions which examine the same learning points as
those in Question 3 above.
Question 4: Find the derivative of 1/x2.
Question 5: Find the derivative of (x4
- x3)/x2.
Either or both of these new questions could have
steps. Question 4 could have the information only step telling the candidates
to express 1/x2 as xp
for example while Question 5 could have the same steps as the original question.
It is possible that there will still be some discrepancies between marks in
certain questions and only by several iterations will a satisfactory resolution
be achieved. However, it is hoped that the number of iterations can be reduced
with the help of the kind of analysis outlined herein.
4. Follow through errors
There was one other way that partial credit could
be gained in Question 1. This was for the incorrect answer to the antiderivative
to be taken into account when evaluating the integral. In this case the candidate
has carried out the correct step, but with an incorrect previous answer.
The partial credit in this case is obtained for what could be called a follow
through error. These occur quite often in paper based examinations and present
even more of a problem when the question asks the candidate to do two or more
tasks each being dependant on a previous answer. Here is a typical example
of such a question.
Question
6: Find the coordinates of the stationary points of the
curve with equation y = x3/3 + x2
- 3x - 11. Using differentiation, determine their nature.
The candidate is set two tasks in this question.
If they get the answer to the first wrong then they are likely to get the
second wrong also even if they have the correct method for determining the
nature of the stationary point and make no mistakes in this process. In a
paper-based examination the marker would award partial credit for the method
used to classify stationary points even if the values the candidates had for
the stationary points are wrong. If the application of follow through is
not possible in an ICT exam, one alternative would be to split such a question
into two as follows.
Question
7: Find the coordinates of the stationary points of the
curve with equation y = x3/3 - 2x2
+ 3x + 1.
Question
8: The curve with equation y
= x3/3 - x2 - 3x + 1
has derivative x2 - 2x - 3
and stationary points at (-1, 8/3) and (3,
-8). Using differentiation, determine the nature of the stationary points.
Note that a different
equation for the curve is used in each question so the answer is not revealed
in the subsequent question.
The same learning points are examined by the two
new questions as the original. The advantage over the original is that even
if the candidate does make errors in finding the stationary points these do
not effect the candidates ability to obtain credit for knowing how to classify
the stationary points.
5. Questions with "show that…"
For some questions
the answer is contained within the question, and the student is required to
show why this is the case, for example,
Question
9: Show that the line with equation y = 2x - 3
is a tangent to the circle with equation x2 + y2
+ 2x - 4 = 0.
Even in a paper-based examination this can be
awkward to mark. A candidate may well write down working unrelated to what
is required and then finish off their solution with "therefore the line
with equation y
= 2x - 3 is a tangent to the circle with x2 + y2 + 2x - 4 = 0."
Even though that is what is required as the last line for the correct solution,
a marker has to look carefully at the whole solution to see how much credit
should be awarded. Since the marks are basically given for the working in
this type of question it is a very awkward question if the assessment is to
be done using ICT. However, even in this type of example there are ways of
assessing the same learning points in an ICT examination. Two possible approaches
are presented in Questions 10 and 11 below.
Question
10: Describe how to show that the line with equation y = 2x - 3 is
a tangent to the circle with equation x2 + y2
+ 2x - 4 = 0.
Find the x-coordinate of the point of intersection of
this line and circle.
The student is first asked to complete the sentence
"To show that this line is a tangent to this circle I would" describing
the process that they would go through to carry out the question. They then
have to do the calculations to find the x-coordinate of the point of intersection
for the specific example given.
In general, it is harder to explain how to do
something rather than just doing it so this revised question may be considerably
more challenging than the original. In addition, the description (which may
turn out to be a mixture of words and mathematical expressions) then has to
be assessed by a human marker unless a suitable free text marking package
is available.
There is an alternative approach to turn this
into an ICT question. While it is not as practical to have random numbers
in questions on paper, this is much easier in an ICT examination. This can
be exploited to change the question from "show that it is a tangent"
into "determine whether it is a tangent" and randomise the equation
of the line or the equation of the circle or both so that the line will only
sometimes be a tangent. So, a candidate might see on screen the following
question:
Question
11: Determine whether the straight line with equation y = 2x + 2
is a tangent to the circle x2 + y2
+ 2x - 4 = 0.
The candidates are asked for the intersection points of the line and the circle
and then asked if the line is a tangent or not. [It is not a tangent for the
numbers given in question 11.]
It could be argued that the second version of
the ICT question is harder as candidates do not have the reassurance in advance
of what the answer is going to be. Nevertheless, in terms of learning points
this may be the closer version of the two ICT questions to the original.
Either of these versions may be a satisfactory
method for similar types of questions but there still remain problems with
questions on mathematical induction, for example, or more generally questions
asking for proofs. Usually, these are not testing the lower order skills
but rather analysis, synthesis or evaluation. The tools of CAA are unable
to measure the higher order skills consistently at the present stage of technology
in subjects like Mathematics though developments through the AIM Project [7,8]
are addressing such issues.
Acknowledgements
The authors are currently engaged on the PASS-IT Project [9] and are grateful
to many colleagues for the support and encouragement during the preparation
of this article.
References
[1] Computer Aided Learning in Mathematics (CALM) website, http://www.calm.hw.ac.uk
[2] Mathwise website, http://www.bham.ac.uk/mathwise
[3] Fiddes, D, Korabinski, A, McGuire, G, Youngson, M and McMillan, D, “Does
the mode of delivery affect Mathematics Examination Results?” Alt-J (10)
p60 – 69. 2002
[4] McGuire G.R. and Youngson, M.A. , “Assessing ICT Assessment in Mathematics”,
http://ltsn.mathstore.ac.uk/articles/maths-caa-series/mar2002/index.shtml,
2002
[5] McGuire, G.R. Youngson, M.A., Korabinski A.A and McMillan, D. (2002)
“Partial credit in mathematics exams - a comparison of traditional and
CAA exams”, Sixth international computer assisted assessment conference,
Loughborough University, 223-230 http://www.lboro.ac.uk/service/ltd/flicaa/conf2002/pdfs/mcguire_gr1.pdf
, 2002.
[6] Beevers, C, Youngson, M, McGuire, G, Wild D and Fiddes, D, “Issues
of Partial Credit in Mathematical Assessment by Computer”, ALT-J vol.
7, 26-32, 1999
[7] Sangwin, C, “Assessing higher skills with computer algebra marking”,
JISC Technology and Standards Watch, 2002
[8] Hermans, D F M, “Intelligent on-line assessment of mathematics using
AIM”, Proceedings of the 6th International Conference on Technology
in Mathematics Teaching, Volos, Greece, October 2003, p.244
[9] PASS-IT project website, http://www.pass-it.org.uk
